Q3Cloud Computing

Parallel programming paradigms divide a computational problem into multiple independent (or partially dependent) sub-tasks that execute simultaneously, typically across multiple processor cores or processors within a single tightly-coupled system (sharing a common memory space), to reduce overall computation time — common parallel programming models include shared-memory multithreading (e.g., using OpenMP or Java threads, where multiple threads within a single process directly access shared memory) and data-parallel programming (applying the same operation simultaneously across many independent data elements, as in GPU/SIMD-based computation).

Distributed programming paradigms coordinate computation across multiple independent, physically separate machines (nodes) connected over a network, each with its own separate memory space, requiring explicit message passing or other inter-node communication mechanisms to coordinate the collective computation and exchange intermediate results — common distributed programming models include Remote Procedure Call (RPC)-based communication, message-passing frameworks (such as MPI, the Message Passing Interface), and, particularly relevant to cloud/big-data computing, the MapReduce programming model.

MapReduce as a Cloud-Relevant Distributed Programming Paradigm

The MapReduce programming model, popularized by Google and implemented as open-source software in Apache Hadoop, is the paradigm most closely associated with practical cloud computing, since it allows a large-scale data-processing task to be automatically distributed across hundreds or thousands of commodity machines in a cloud data center, with the underlying framework handling task scheduling, data partitioning, fault tolerance (automatically re-executing failed tasks on other available nodes), and result aggregation, largely hidden from the application programmer.

A MapReduce job comprises two main phases: the Map phase, in which the input dataset (typically split into many independent chunks distributed across the cluster's storage nodes) is processed in parallel by many map tasks, each transforming its assigned input chunk into a set of intermediate key-value pairs; and the Reduce phase, in which all intermediate values associated with the same key (which may have been produced by different map tasks running on different nodes) are gathered together (a step called 'shuffle and sort') and processed by reduce tasks, which combine/aggregate these values into the final output result for that key.

Practical Example: Word Count

A classic illustrative MapReduce example is counting the frequency of each distinct word across a very large collection of text documents distributed across a cluster: the Map function is applied independently, in parallel, to each document (or chunk of documents) on whichever node currently holds that data, emitting an intermediate key-value pair (word, 1) for every word occurrence encountered; the framework then automatically groups together all intermediate pairs sharing the same word (key) from across all the map tasks; and the Reduce function, running for each distinct word, simply sums up all the '1' values associated with that word to produce the final total occurrence count for that word across the entire, massive document collection — a computation that would be prohibitively slow to perform on a single machine for a truly massive dataset, but which the MapReduce paradigm allows to be completed efficiently by harnessing the combined processing power of an entire cluster of cloud-provisioned machines working in parallel, with the framework itself managing the complex coordination, fault-tolerance, and data-movement logistics required to make this distributed parallel execution transparent to the application developer, who need only write the relatively simple Map and Reduce functions themselves.