Q19Big Data Analytics
Question
Detail the complete MapReduce execution flow.
Answer
The MapReduce execution flow encompasses input splitting, mapping, shuffling/sorting, reducing, and output generation.
The complete flow of a MapReduce job:
1. InputFormat: Validates the input format and splits the input data into logical InputSplits. It provides a RecordReader to extract key-value pairs.
2. Map Phase: The RecordReader passes key-value pairs to the map() function. The Mapper processes them and emits intermediate key-value pairs.
3. Combiner (Optional): Acts as a mini-reducer to aggregate data locally on the mapper node, reducing network traffic.
4. Partitioner: Determines which reducer will process a specific key. It ensures all values for a single key go to the same reducer.
5. Shuffle and Sort: The framework fetches intermediate outputs from mappers over the network, then sorts and merges them by key.
6. Reduce Phase: The reduce() function receives the key and an iterator over its values, processes them, and emits final key-value pairs.
7. OutputFormat: Writes the final key-value pairs to HDFS.
