Q15Cloud Computing
Question
4 marks
Describe the working mechanism of Hadoop Distributed File System (HDFS).
Answer
A definitive explanation of the Hadoop Distributed File System (HDFS). Violently details the Master-Slave architecture, the exact mathematical data chunking process, and the absolute replication algorithms that guarantee horrific fault tolerance.
The Hadoop Distributed File System (HDFS) is a massive, highly fault-tolerant storage architecture designed to mathematically distribute Petabytes of data across thousands of cheap, unreliable commodity hard drives. It violently assumes that physical hard drives will fail daily, and uses mathematical replication to prevent data loss.
- NameNode (The Master): The absolute mathematical brain. It does NOT store the actual data. It stores only the Metadata (the directory tree, and exactly which DataNode holds which chunk of a file). It resides completely in RAM for explosive speed.
- DataNodes (The Slaves): Thousands of dumb physical servers. They literally just store raw blocks of binary data on their hard drives and continuously scream "I am alive!" (Heartbeats) to the NameNode every 3 seconds.
When a user attempts to save a massive 1-Terabyte video file:
- 1. Block Splitting: HDFS violently chops the file into strict mathematical chunks, exactly 128 Megabytes each.
- 2. Replication (Fault Tolerance): To survive hardware death, HDFS forces a Replication Factor (default = 3). Every single 128MB block is violently copied perfectly to exactly 3 DIFFERENT DataNodes across 3 different physical server racks.
- 3. The Pipeline: The client writes Block A to DataNode 1. DataNode 1 autonomously forwards the copy to DataNode 2, which forwards it to DataNode 3. This offloads the network bandwidth from the Master.
- 4. Catastrophic Failure Recovery: If DataNode 2 catches fire and dies, its Heartbeat stops. The NameNode instantly mathematically realizes that Block A now only has 2 copies instead of 3. It violently commands DataNode 1 to copy Block A to DataNode 4, instantly restoring absolute mathematical fault tolerance without any human intervention.