High-Performance Persistent Storage System for BigData Analysis

Inhaltsverzeichnis (Table of Contents)

• Chapter 1
  INTRODUCTION
• Chapter 2 - LITRATURE REVIEW
  
  • 2.1 Advantages of using Hadoop
  • 2.2 Big Data
  • 2.3 Project Architecture
  • 2.4 Goal: HDFS-HMFGR
  • 2.5 INTERCONNECT TECHNOLOGIES (Hardware Solution)
    
    • 2.5.1 Traditional Interconnect 10GigE Network
    • 2.5.2 Infiniband Technology
      
      • 2.5.2.1 IPOIB Interconnect Technology
      • 2.5.2.2 RDMA-IB Interconnect Technology
    • 2.6 Memory Allocation with MemCached (Software Solution)
• Chapter 3 - Experimental Testbed System
  
  • 3.1 An Insight into SSD and HDD
• Chapter 4 - Installation, Designing and Implementation of System
  
  • 4.1 Set a LAN and System service of Network
  • 4.2 Hadoop Installation Guide
    
    • 4.2.1 Steps On each Machine
      
      • 4.2.1.1 Install prerequisites
      • 4.2.1.2 Adding a dedicated Hadoop system user
      • 4.2.1.3 Setup hostname
    • 4.2.2 Steps On Master
      
      • 4.2.2.1 Install Hadoop
      • 4.2.2.2 Configuration ssh
      • 4.2.2.3 Install NFS
      • 4.2.2.4 Configuration
    • 4.2.3. Run Hadoop
      
      • 4.2.3.1 Run HDFS
      • 4.2.3.2 Rum Map Reduce Job
      • 4.2.3.3 Run All Daemon
      • 4.2.3.4 Hadoop Web Interfaces
    • 4.3 Enable Ethernet and InfiniBand (SR-IOV)
      
      • 4.3.1 Enable Ethernet SR-IOV
      • 4.3.2 Enable InfiniBand SR-IOV
    • 4.4 Steps to setup IPOIB on Quanta machines
    • 4.5 Install MemCached on CentOS
  • Chapter 5 - Benchmarking
    
    • 5.1 ATTO Disk Benchmarking
    • 5.2 HD Tune Pro:
    • 5.3 Linux Disk Utilities:
    • 5.4 HiBench (Hadoop Benchmarking Suit)
      
      • 5.4.1 Micro-Benchmarks
        
        • 5.4.1.1 Sort
        • 5.4.1.2 Word Count
        • 5.4.1.3 TeraSort
        • 5.4.1.4 Enhanced DFSIO
      • 5.4.2 Web Search:
        
        • 5.4.2.1 Nutch Indexing
        • 5.4.2.2 Page Ranking
      • 5.4.3 Machine Learning:
        
        • 5.4.3.1 Bayesian Classification
        • 5.4.3.2 K-means clustering
      • 5.4.4 Analytical Query
        
        • 5.4.4.1 Hive Join
        • 5.4.4.2 Hive Aggregation
  • Chapter 6 - Performance Evaluation of SSD and HDD on Hadoop using 10 GigE
    
    • 6.1 Sort Work Load:
    • 6.2 Word Count Work Load
    • 6.3 Tera Sort Work Load:
  • Chapter 7 – Performance Evaluation of SSD and HDD on Hadoop using IPOIB

  Zielsetzung und Themenschwerpunkte (Objectives and Key Themes)

  The dissertation aims to analyze and characterize the performance of external sorting algorithms within the Hadoop (MapReduce) framework using Solid State Drives (SSDs) and Hard Disk Drives (HDDs) connected through various interconnect technologies. The research explores the potential for upgrading traditional servers and cloud systems to handle big data workloads effectively by leveraging modern storage devices and networking systems. The dissertation seeks to demonstrate that such upgrades can enhance performance without significant costs, reducing power consumption and latency while improving throughput. Key themes include:
  • Performance evaluation of Hadoop with SSD and HDD
  • Impact of interconnect technologies on Hadoop performance
  • Optimization of Hadoop system using modern storage devices and networking systems
  • Cost-effective performance enhancement for traditional servers and cloud systems
  • Reducing power consumption and latency while increasing throughput

  Zusammenfassung der Kapitel (Chapter Summaries)

  • Chapter 1: INTRODUCTION
  • Chapter 2: LITRATURE REVIEW
    This chapter presents a comprehensive overview of Hadoop's advantages, the concept of big data, and the project's architecture. It also delves into various interconnect technologies, including 10GigE, IPOIB, and RDMA-IB, as well as the role of MemCached for memory allocation.
  • Chapter 3: Experimental Testbed System
    Chapter 3 provides an in-depth analysis of Solid State Drives (SSDs) and Hard Disk Drives (HDDs) as storage devices.
  • Chapter 4: Installation, Designing and Implementation of System
    This chapter describes the detailed process of installing and configuring the Hadoop system, including network setup, Hadoop installation steps on individual machines and the master node, and enabling Ethernet and InfiniBand connectivity. It also covers the setup of IPOIB on Quanta machines and MemCached installation on CentOS.
  • Chapter 5: Benchmarking
    This chapter outlines the methodology employed for benchmarking the system, utilizing tools like ATTO Disk Benchmarking, HD Tune Pro, Linux Disk Utilities, and HiBench (Hadoop Benchmarking Suit). The chapter delves into the various micro-benchmarks, web search applications, machine learning algorithms, and analytical queries used for evaluating the system's performance.
  • Chapter 6: Performance Evaluation of SSD and HDD on Hadoop using 10 GigE
    This chapter presents the performance evaluation of SSD and HDD in the Hadoop environment using 10GigE connectivity, analyzing the results of Sort, Word Count, and Tera Sort workloads.
  • Chapter 7: Performance Evaluation of SSD and HDD on Hadoop using IPOIB
    This chapter investigates the performance of SSD and HDD in the Hadoop environment using IPOIB technology, providing insights into the performance gains achieved through this interconnect method.

  Schlüsselwörter (Keywords)

  The dissertation primarily focuses on the performance of Hadoop with SSD and HDD utilizing various interconnect technologies, including 10GigE, IPOIB, and RDMA-IB. The key concepts revolve around optimizing Hadoop systems through modern storage devices and networking solutions, aiming to enhance performance, reduce power consumption, and improve throughput. The research explores the possibility of upgrading traditional servers and cloud systems to handle big data workloads efficiently without significant cost implications.

  Frequently Asked Questions

  What is HiBench?

  HiBench is a comprehensive benchmark suite for Hadoop that includes micro-benchmarks and real-time applications to evaluate system performance.

  How do SSDs compare to HDDs in Big Data analysis?

  The research analyzes how Solid State Drives (SSDs) significantly improve throughput and reduce latency compared to traditional Hard Disk Drives (HDDs) in Hadoop environments.

  What are interconnect technologies like RDMA-IB?

  These are high-speed networking technologies (Remote Direct Memory Access over InfiniBand) that allow for faster data transfer between nodes in a cluster.

  Can old cloud systems be upgraded for Big Data?

  Yes, the study demonstrates that traditional servers can be upgraded with modern storage and interconnects to perform on par with modern systems at a lower cost.

  What role does MemCached play?

  MemCached is used as a software solution for efficient memory allocation to optimize the processing of large datasets.