Q13Cloud Computing

The Hadoop Distributed File System (HDFS) is designed to store very large files across machines in a large cluster. The NameNode acts as the master, managing the file system namespace, maintaining the metadata (like directory structure and file-to-block mapping), and regulating access. The DataNodes act as slaves, storing and retrieving the actual data blocks, which are replicated across multiple nodes (typically 3) for fault tolerance. They report block health and status to the NameNode periodically.

Key Components and Mechanisms

• Blocks: Files are split into large fixed-size blocks (default 128 MB) so that a single file can be spread across many machines and processed in parallel.
• NameNode Metadata: The NameNode keeps the entire namespace image and block map in memory for fast lookups, and persists it to disk via the FsImage and EditLog for recovery after a restart.
• Heartbeats and Block Reports: Each DataNode sends periodic heartbeats to prove it is alive and block reports listing the blocks it holds; if a heartbeat is missed, the NameNode marks the DataNode dead and schedules re-replication of its blocks elsewhere.
• Secondary NameNode: Periodically merges the EditLog into the FsImage to prevent the log from growing unbounded, reducing NameNode restart time.
• Client Read/Write Path: A client first contacts the NameNode to get block locations, then streams data directly to or from the relevant DataNodes, keeping the NameNode out of the data path and preventing it from becoming a bottleneck.

This design gives HDFS high throughput for large sequential reads and strong fault tolerance, but it makes the NameNode a single point of metadata failure, which production clusters mitigate using NameNode HA with a standby node and shared edit logs.