Q19Big Data Analytics
Question
Explain the Lambda architecture for big data systems.
Answer
An exhaustive mathematical dissection of the Lambda Architecture. Details how Nathan Marz violently solved the conflict between slow batch processing (Hadoop) and inaccurate real-time streaming (Storm) by fusing them into an impenetrable, immutable three-layer system.
Before the Lambda Architecture, Big Data systems faced a catastrophic paradox. If you used Hadoop MapReduce, you got absolute mathematical accuracy, but the report took 8 hours to generate (useless for real-time fraud detection). If you used Apache Storm (streaming), you got sub-second speed, but the data was often inaccurate due to dropped packets. Nathan Marz invented the Lambda Architecture to violently fuse these two paradigms together, guaranteeing both absolute accuracy AND microsecond latency.
Lambda operates on one rigid physical law: Data is completely immutable. Once a record is written, it can NEVER be updated or deleted. If a user changes their address, you do not UPDATE the database; you strictly append a new record with a new timestamp. This completely destroys database lock contention and race conditions.
1. The Batch Layer (The Source of Truth)
This layer uses Hadoop HDFS and MapReduce/Spark.
- Function: It stores the absolute master dataset. Every single byte of data ever generated is dumped here.
- Execution: Every few hours, it violently processes the ENTIRE dataset from the beginning of time to mathematically compute massive "Batch Views" (pre-calculated answers).
- Merit: Absolute mathematical perfection. It resolves all late-arriving or out-of-order data. Demerit: It is hours out of date.
2. The Speed Layer (The Real-Time Engine)
This layer uses Apache Kafka and Spark Streaming or Storm.
- Function: It only looks at the exact data that has arrived since the last Batch Layer run began.
- Execution: It violently processes this tiny slice of data in milliseconds, generating "Real-time Views".
- Merit: Instantaneous analytics. Demerit: It is mathematically approximate. It might drop a packet or double-count a record due to network chaos.
3. The Serving Layer (The Fusion Interface)
This layer uses ultra-fast NoSQL databases like Apache Cassandra or HBase.
- Function: It loads the massive Batch Views and the tiny Real-time Views into RAM.
- Execution: When the user queries the dashboard, the Serving Layer mathematically merges the perfect (but old) Batch View with the chaotic (but new) Real-time View.
- The Result: The user receives a query response in 10 milliseconds that is mathematically perfectly accurate up to a few hours ago, and highly accurate up to the exact current millisecond.