Q14Cloud Computing
Question
Discuss data replication in HDFS.
Answer
A deep architectural dive into the Hadoop Distributed File System (HDFS), detailing how its aggressive 3x Data Replication strategy mathematically guarantees absolute fault tolerance and data survival against catastrophic hardware failure.
The Hadoop Distributed File System (HDFS) is designed to run on a massive cluster of cheap, unreliable commodity hardware. In a cluster of 10,000 servers, hard drive failure is not a probability; it is a mathematical certainty occurring daily. To prevent catastrophic data loss, HDFS implements a highly aggressive, default 3x Data Replication strategy.
The Mathematical Mechanics of Replication
- Block Splitting: When a massive 1GB file is uploaded to HDFS, the system violently chops it into massive, rigid mathematical chunks called Blocks (default size is 128 MB).
- The Replication Pipeline: The central NameNode commands the client to write the block to a specific DataNode. 1. The client writes Block A to DataNode 1. 2. DataNode 1 immediately opens a network pipeline and aggressively copies Block A to DataNode 2. 3. DataNode 2 simultaneously pipelines the data to DataNode 3.
- Rack Awareness Algorithm: HDFS does not place replicas randomly. It executes a strict mathematical Rack Awareness protocol to survive catastrophic power grid failures. - Replica 1 is placed on a server in Rack A. - Replica 2 is placed on a different server in the exact same Rack A. - Replica 3 is violently forced onto a completely different physical Rack B. If Rack A loses power completely, the data perfectly survives on Rack B.
Heartbeats and Self-Healing
Every DataNode sends a mathematical "Heartbeat" ping to the NameNode every 3 seconds. If a node fails to ping, the NameNode mathematically calculates exactly which blocks were destroyed. It instantly commands surviving nodes to massively clone the remaining replicas, violently restoring the strict 3x replication factor to heal the cluster.