Q21Big Data Analytics

Writing a raw Java MapReduce program to execute a simple JOIN between two massive 5-Terabyte datasets requires roughly 300 lines of highly complex, error-prone Java code. It is an architectural nightmare. Yahoo invented Apache Pig to violently destroy this complexity. Pig is a high-level Dataflow programming platform that allows analysts to write 5 lines of simple script, which the Pig architecture mathematically compiles into the 300 lines of optimized Java MapReduce.

Pig Latin is not SQL. SQL is a declarative language (you tell the system what you want, and the system figures out how to do it). Pig Latin is a Dataflow language. The programmer mathematically dictates the exact step-by-step physical transformation of the data.

• A = LOAD "data.txt" AS (id:int, name:chararray); (Ingest data from HDFS)
• B = FILTER A BY id > 100; (Violently strip out useless records)
• C = GROUP B BY name; (Mathematically group the data, triggering a massive network shuffle)
• D = FOREACH C GENERATE group, COUNT(B); (Calculate the aggregation)
• STORE D INTO "output"; (Trigger the physical MapReduce execution)

When a user submits a Pig script, it is not executed immediately. It passes through a rigid mathematical compilation pipeline:

• Parser: Validates the absolute syntax of the Pig Latin script and verifies the data types. It generates a Directed Acyclic Graph (DAG) representing the logical dataflow.
• Optimizer: The absolute genius of Pig. The Logical Optimizer mathematically inspects the DAG. If the user wrote a JOIN followed by a FILTER, the optimizer violently rearranges the code, pushing the FILTER before the JOIN. This mathematically drops 90% of the data before the network shuffle, reducing processing time from hours to minutes.
• Compiler: It takes the optimized logical DAG and mathematically translates it into a physical sequence of actual MapReduce jobs (e.g., Job 1: Map-only. Job 2: Map and Reduce).
• Execution Engine: It blindly submits the compiled .jar files to the Hadoop cluster (YARN) and monitors the catastrophic execution, handling failures automatically.

Pig utilizes a highly flexible, nested mathematical data model, completely unlike rigid SQL tables:

• Atom: A single primitive value (e.g., an Integer 5).
• Tuple: An ordered set of fields, exactly like a row in SQL (e.g., (5, "John")).
• Bag: An unordered collection of tuples. Crucially, a Bag can be placed inside another Tuple, mathematically allowing infinite nested hierarchies of data, which is completely impossible in standard SQL.