Q22Big Data Analytics
Question
Detail the Spark architecture and RDD operations.
Answer
Spark architecture involves a Driver Program, Cluster Manager, and Worker Nodes operating on RDDs via Transformations and Actions.
Apache Spark is a unified analytics engine for large-scale data processing. Its architecture operates in a master-worker fashion:
1. Driver Program: The process that runs the main() function of the application and creates the SparkContext. It converts the user's code into tasks and schedules them on executors.
2. Cluster Manager: A pluggable component (like YARN, Mesos, or Spark's Standalone Manager) responsible for acquiring resources on the cluster.
3. Worker Nodes: Nodes that run application code in the cluster. They host Executors, which are processes that run tasks and keep data in memory or disk storage.
RDD (Resilient Distributed Dataset) operations are divided into two types:
- Transformations: Lazy operations that create a new RDD from an existing one (e.g., map, filter, flatMap). They do not compute their results immediately.
- Actions: Operations that return a value to the driver program or write data to storage, triggering the execution of transformations (e.g., reduce, collect, count).
