RTUComputer ScienceYr 2021 · Sem 72021

Q20Cloud Computing

Question

10 marks

Describe the MapReduce programming model in detail. How does it handle fault tolerance? Give an example of a MapReduce job.

Answer

A definitive theoretical breakdown of the MapReduce programming model. Violently details the split of computational logic into Map and Reduce phases, the underlying Shuffle and Sort mechanisms, and the absolute hardware fault tolerance achieved through aggressive task reassignment.

Invented by Google to index the entire internet, MapReduce is a mathematically aggressive programming architecture designed to process Petabytes of data across thousands of commodity servers. It completely abstracts the horrors of parallel processing, network communication, and hardware failure away from the programmer. The programmer only writes two rigid mathematical functions: Map() and Reduce().

  • 1. Input Splitting: The master node violently chops a massive 1-Terabyte dataset into thousands of independent 128MB chunks.
  • 2. The Map Phase (Parallel Execution): The master dispatches the user's Map() code to thousands of worker nodes simultaneously. Each worker executes the code on its specific 128MB chunk. Mathematics: The Map function takes raw text and outputs intermediate Key-Value pairs. Map(String key, String value) -> list(Key, Value)
  • 3. Shuffle and Sort (The Network Bottleneck): The absolute most complex phase. The Hadoop framework takes all millions of intermediate Key-Value pairs from all workers, violently sorts them by Key, and routes all identical Keys to the exact same Reduce worker across the network.
  • 4. The Reduce Phase (Aggregation): A Reduce worker receives a specific Key and a massive list of all values associated with that key. It mathematically aggregates them into a single final output. Reduce(String key, list(Values)) -> list(Key, Value)

MapReduce violently assumes hardware will die during processing. It achieves fault tolerance through Master Surveillance:

  • Every worker node continuously blasts a "Heartbeat" signal to the Master Node over the network.
  • If a worker CPU overheats and dies midway through a massive Map task, the heartbeat stops.
  • The Master Node mathematically detects the silence, instantly declares the worker dead, and violently reschedules that exact Map task to a brand new, healthy worker node. The programmer is completely unaware this horrific failure even occurred.

Goal: Count the occurrences of every word in a 10TB library of books.

  • Input: ["apple banana apple"]
  • Map Function: Emits a 1 for every word it sees. ("apple", 1), ("banana", 1), ("apple", 1)
  • Shuffle/Sort: Routes all "apple"s to Reducer A. ("apple", [1, 1]), ("banana", [1])
  • Reduce Function: Mathematically sums the list. ("apple", 2), ("banana", 1)
Back to Paper