Q20Cloud Computing

Apache Hadoop is the absolute foundational open-source architecture for Big Data Cloud computing. It was engineered to solve a catastrophic mathematical problem: you cannot store or process a 10-Petabyte dataset on a single server. Hadoop violently shatters the data and the processing logic, distributing them across a massive cluster of thousands of cheap, unreliable commodity computers. The ecosystem is rigidly divided into two core pillars: HDFS (Storage) and MapReduce (Processing).

HDFS operates on a strict Master-Slave architecture.

The NameNode (The Master)

The absolute brain of the file system. It is a single, massive server that stores absolutely zero actual user data. It strictly stores the Metadata (the massive mathematical index of the file system) entirely in its RAM for blisteringly fast access.

• It mathematically tracks the exact physical location of every single 128MB block across the 10,000 servers.
• It aggressively monitors the Heartbeats of DataNodes. If a node dies, it instantly orchestrates the replication of the lost blocks to maintain the strict 3x fault-tolerance.

The DataNode (The Slave)

The thousands of "dumb" worker servers. They physically store the actual binary blocks of data on their local magnetic hard drives. They blindly follow the mathematical commands of the NameNode to read, write, or violently delete blocks.

In traditional architectures, 10TB of data is transmitted over the network to the CPU. In Hadoop, this would cause a catastrophic network meltdown. MapReduce utilizes "Data Locality": it mathematically transmits the tiny Java code strictly to the physical server where the data already resides.

The JobTracker (The Master)

The supreme commander of processing. When a user submits a massive MapReduce job, the JobTracker communicates with the NameNode to locate exactly which DataNodes hold the required blocks. It then violently shatters the massive job into thousands of microscopic "Map" and "Reduce" tasks, and intelligently assigns them to the specific nodes holding the data.

The TaskTracker (The Slave)

A background daemon running on every single DataNode. It physically receives the Java code from the JobTracker and aggressively executes it against the local data blocks. It constantly sends mathematical progress reports back to the JobTracker. If a TaskTracker crashes mid-calculation, the JobTracker instantly detects the failure and violently re-assigns the task to another node holding a replica of the data.

Evolution to YARN

In modern Hadoop 2.x and later, the monolithic JobTracker was replaced by YARN (Yet Another Resource Negotiator), splitting its responsibilities into a ResourceManager (global resource allocation across the cluster) and a per-job ApplicationMaster (task scheduling and monitoring for a single job), while the TaskTracker was replaced by a generic NodeManager. This decoupling of resource management from job scheduling was necessary because the original JobTracker became both a scalability bottleneck and a single point of failure as clusters grew beyond a few thousand nodes, and it also allows non-MapReduce frameworks like Spark and Tez to share the same Hadoop cluster, since YARN is a general-purpose resource scheduler rather than one tied exclusively to the MapReduce execution model.