{"pinia":{"ui":{"prefs":{"theme":"system","textScale":"md","fontFamily":"sans","reducedMotion":false,"highContrast":false}},"academy":{"universities":[],"universitySummaries":[{"slug":"rtu","name":"RTU","fullName":"Rajasthan Technical University","branches":[{"slug":"first-year-common","name":"First Year (Common)"},{"slug":"computer-science","name":"Computer Science"},{"slug":"mechanical","name":"Mechanical"},{"slug":"civil","name":"Civil"},{"slug":"ai-data-science","name":"AI & Data Science"},{"slug":"ee-ec-eee","name":"EE / EC / EEE"}]},{"slug":"aktu","name":"AKTU","fullName":"Dr. A. P. J. Abdul Kalam Technical University","branches":[{"slug":"first-year-common","name":"First Year (Common)"},{"slug":"computer-science","name":"Computer Science"}]}],"activePapers":[{"id":"big-data-analytics-2022","title":"Big Data Analytics (2022)","subjectSlug":"big-data-analytics","subjectName":"Big Data Analytics","universitySlug":"rtu","branchSlug":"computer-science","year":4,"semester":8,"examYear":2022,"paperCode":"8E1809","questions":[{"id":"v2-big-data-analytics-2022-pA-q1","number":1,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"Define Big Data."}],"answer":{"summary":"Big Data refers to vast and complex datasets that traditional data processing applications cannot handle.","explanation":[{"type":"paragraph","content":"It is characterized by extreme volume, high velocity of generation, and wide variety of data types, requiring specialized distributed systems like Hadoop or Spark for storage and analysis."}]}},{"id":"v2-big-data-analytics-2022-pA-q2","number":2,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is HDFS?"}],"answer":{"summary":"HDFS is the Hadoop Distributed File System designed to store very large files across machines in a large cluster.","explanation":[{"type":"paragraph","content":"It provides high-throughput access to application data and is highly fault-tolerant. HDFS stores data in large blocks (typically 128MB) and replicates them across multiple nodes to ensure data reliability."}]}},{"id":"v2-big-data-analytics-2022-pA-q3","number":3,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"Name two NoSQL databases."}],"answer":{"summary":"Two popular NoSQL databases are MongoDB and Apache Cassandra.","explanation":[{"type":"paragraph","content":"MongoDB is a document-oriented NoSQL database that stores data in JSON-like format. Apache Cassandra is a wide-column store designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure."}]}},{"id":"v2-big-data-analytics-2022-pA-q4","number":4,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is a NameNode?"}],"answer":{"summary":"The NameNode is the master node in HDFS that manages the file system namespace and controls access to files by clients.","explanation":[{"type":"paragraph","content":"It maintains the directory tree of all files in the file system and tracks where across the cluster the file data is kept. It does not store the actual data; that is the job of the DataNodes."}]}},{"id":"v2-big-data-analytics-2022-pA-q5","number":5,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"Define RDD in Spark."}],"answer":{"summary":"RDD (Resilient Distributed Dataset) is the fundamental data structure of Apache Spark.","explanation":[{"type":"paragraph","content":"It is an immutable distributed collection of objects that can be processed in parallel. RDDs offer fault tolerance through lineage graphs, automatically recovering lost data resulting from node failures."}]}},{"id":"v2-big-data-analytics-2022-pA-q6","number":6,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is the purpose of YARN?"}],"answer":{"summary":"YARN (Yet Another Resource Negotiator) is Hadoop's cluster resource management system.","explanation":[{"type":"paragraph","content":"Introduced in Hadoop 2, YARN separates the resource management and processing components. It allocates system resources to the various applications running in a Hadoop cluster and schedules tasks to be executed on different cluster nodes."}]}},{"id":"v2-big-data-analytics-2022-pA-q7","number":7,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"State the CAP theorem."}],"answer":{"summary":"The CAP theorem states that a distributed system can only provide two of three guarantees: Consistency, Availability, and Partition tolerance.","explanation":[{"type":"paragraph","content":"Formulated by Eric Brewer, it implies that in the presence of a network partition (P), one has to choose between consistency (C) and availability (A). Most NoSQL databases are designed with specific trade-offs according to this theorem."}]}},{"id":"v2-big-data-analytics-2022-pA-q8","number":8,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is Hive?"}],"answer":{"summary":"Apache Hive is a data warehouse software project built on top of Apache Hadoop for providing data query and analysis.","explanation":[{"type":"paragraph","content":"It gives an SQL-like interface (HiveQL) to query data stored in various databases and file systems that integrate with Hadoop. It abstracts the complexity of MapReduce by converting queries into MapReduce, Tez, or Spark jobs."}]}},{"id":"v2-big-data-analytics-2022-pA-q9","number":9,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"Define Clustering in Big Data context."}],"answer":{"summary":"Clustering is an unsupervised machine learning technique used to group similar data points together.","explanation":[{"type":"paragraph","content":"In big data, it helps in customer segmentation, pattern recognition, and anomaly detection by identifying inherent groupings in large datasets without prior labels. K-Means is a common clustering algorithm."}]}},{"id":"v2-big-data-analytics-2022-pA-q10","number":10,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is Pig Latin?"}],"answer":{"summary":"Pig Latin is a high-level data flow scripting language used with Apache Pig to analyze large datasets.","explanation":[{"type":"paragraph","content":"It abstracts the programming complexity of MapReduce. Pig Latin scripts are translated into a series of MapReduce jobs by the Pig execution engine, making it easier for data engineers to write complex data transformations."}]}},{"id":"v2-big-data-analytics-2022-pB-q1","number":11,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Explain MapReduce architecture."}],"answer":{"summary":"MapReduce architecture involves a Master node managing jobs and Worker nodes executing Map and Reduce tasks.","explanation":[{"type":"paragraph","content":"MapReduce distributes processing across a cluster. The architecture has a JobTracker (or ResourceManager in YARN) that coordinates tasks, and TaskTrackers (NodeManagers) that execute them. In the Map phase, input data is processed to generate intermediate key-value pairs. During the Shuffle and Sort phase, these pairs are grouped by key. Finally, the Reduce phase aggregates these grouped values to produce the final output."}]}},{"id":"v2-big-data-analytics-2022-pB-q2","number":12,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Differentiate between Relational and NoSQL databases."}],"answer":{"summary":"Relational databases use structured schemas and SQL, while NoSQL databases handle unstructured data with flexible schemas.","explanation":[{"type":"paragraph","content":"Relational databases (RDBMS) are table-based, highly structured, and strictly adhere to ACID properties, making them suitable for complex queries. NoSQL databases can be document-based, key-value pairs, graph, or wide-column stores. They offer horizontal scalability, flexible data models, and are better suited for hierarchical data and large-scale big data applications."}]}},{"id":"v2-big-data-analytics-2022-pB-q3","number":13,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Discuss HDFS block replication strategy."}],"answer":{"summary":"HDFS replicates each data block multiple times (default 3) across different DataNodes to ensure fault tolerance.","explanation":[{"type":"paragraph","content":"When a file is written to HDFS, it is split into blocks. The NameNode determines the replication strategy: the first replica is placed on the local node where the client is running, the second replica is placed on a different rack to protect against rack failure, and the third replica is placed on a different node within the same rack as the second. This strategy balances reliability and read/write bandwidth."}]}},{"id":"v2-big-data-analytics-2022-pB-q4","number":14,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Compare Pig Latin with SQL."}],"answer":{"summary":"Pig Latin is a procedural data flow language, whereas SQL is a declarative query language.","explanation":[{"type":"paragraph","content":"SQL focuses on what data to retrieve without specifying how to retrieve it, making it ideal for structured data and analytical queries. Pig Latin is procedural; the programmer defines a series of steps (data flow) to transform data. Pig is better suited for unstructured data preparation, ETL processes, and complex sequential processing where intermediate states are important."}]}},{"id":"v2-big-data-analytics-2022-pB-q5","number":15,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Explain Spark core components."}],"answer":{"summary":"Spark Core is the underlying execution engine of Apache Spark, providing distributed task dispatching, scheduling, and basic I/O functionalities.","explanation":[{"type":"paragraph","content":"It provides the RDD (Resilient Distributed Dataset) abstraction, which allows data to be partitioned across machines. Spark Core is responsible for memory management, fault recovery, and interacting with storage systems. Other libraries like Spark SQL, Spark Streaming, MLlib, and GraphX are built on top of Spark Core to provide specialized data processing capabilities."}]}},{"id":"v2-big-data-analytics-2022-pB-q6","number":16,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Describe the Cassandra data model."}],"answer":{"summary":"Cassandra uses a wide-column store data model optimized for fast writes and highly available distributed storage.","explanation":[{"type":"paragraph","content":"Data is partitioned across the cluster using a Partition Key, and rows within a partition are ordered by Clustering Columns. The schema consists of Keyspaces (similar to a database), Column Families (similar to tables), and Rows containing multiple columns. It allows dynamic column addition and provides decentralized, masterless architecture for high availability."}]}},{"id":"v2-big-data-analytics-2022-pB-q7","number":17,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"How does K-Means clustering work on large datasets?"}],"answer":{"summary":"K-Means groups data into K distinct clusters by iteratively updating cluster centroids to minimize the variance within each cluster.","explanation":[{"type":"paragraph","content":"For large datasets, distributed frameworks like Spark MLlib implement K-Means. Initially, K centroids are chosen randomly. In the Map phase, each data point is assigned to the nearest centroid. In the Reduce phase, new centroids are calculated as the mean of all points assigned to that cluster. This process iterates until the centroids stabilize or a maximum number of iterations is reached."}]}},{"id":"v2-big-data-analytics-2022-pC-q1","number":18,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Describe Hadoop architecture in detail."}],"answer":{"summary":"Hadoop's architecture consists of HDFS for distributed storage and MapReduce/YARN for distributed processing.","explanation":[{"type":"paragraph","content":"The Apache Hadoop architecture is designed to handle massive volumes of data across clusters of commodity hardware. It consists of four main modules: \n\n1. **Hadoop Common**: The core libraries and utilities that support the other modules.\n2. **HDFS (Hadoop Distributed File System)**: A Java-based file system that provides scalable and reliable data storage. It operates on a Master/Slave architecture, with a single NameNode managing the file system metadata and multiple DataNodes storing the actual data blocks. HDFS replicates blocks to prevent data loss.\n3. **MapReduce**: A software framework for easily writing applications that process vast amounts of data in parallel on large clusters in a fault-tolerant manner. \n4. **YARN (Yet Another Resource Negotiator)**: The framework responsible for cluster resource management and job scheduling. It decouples the resource management capabilities from the processing model, allowing multiple processing engines (like Spark and Tez) to run alongside MapReduce."},{"type":"svg","src":"/assets/diagrams/rtu/cs/4/big-data-analytics/diagram-18.svg","alt":"Hadoop Ecosystem and Architecture Diagram","caption":"Hadoop Ecosystem and Architecture Diagram"}]}},{"id":"v2-big-data-analytics-2022-pC-q2","number":19,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Explain YARN architecture and its components."}],"answer":{"summary":"YARN architecture relies on a central ResourceManager, node-level NodeManagers, and per-application ApplicationMasters.","explanation":[{"type":"paragraph","content":"YARN (Yet Another Resource Negotiator) improves upon Hadoop 1.x by separating resource management from job scheduling. Its primary components include:\n\n1. **ResourceManager (RM)**: The master daemon running on a master node. It arbitrates resources among competing applications in the cluster. It contains two main parts: a Scheduler (allocates resources based on capacity/queues) and an ApplicationsManager (accepts job submissions and negotiates the first container to start the ApplicationMaster).\n2. **NodeManager (NM)**: The slave daemon running on every compute node. It is responsible for launching application containers, monitoring their resource usage (CPU, memory, disk, network), and reporting this to the ResourceManager.\n3. **ApplicationMaster (AM)**: A per-application framework-specific entity. It negotiates resources from the ResourceManager and works with the NodeManager(s) to execute and monitor component tasks.\n4. **Container**: Represents a fraction of the cluster's capacity (CPU, memory) allocated by the RM to an application."},{"type":"svg","src":"/assets/diagrams/rtu/cs/4/big-data-analytics/diagram-19.svg","alt":"YARN Architecture and Components Diagram","caption":"YARN Architecture and Components Diagram"}]}},{"id":"v2-big-data-analytics-2022-pC-q3","number":20,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Discuss the Apriori algorithm for market basket analysis."}],"answer":{"summary":"The Apriori algorithm is used for mining frequent itemsets and learning association rules over transactional databases.","explanation":[{"type":"paragraph","content":"Market Basket Analysis aims to identify items that frequently co-occur in transactions (e.g., milk and bread). The Apriori algorithm uses a level-wise search to find these frequent itemsets.\n\n1. **Support**: The frequency of an itemset in the dataset. Apriori sets a minimum support threshold.\n2. **Confidence**: The likelihood that item B is purchased given that item A is purchased.\n3. **Apriori Principle**: If an itemset is frequent, then all of its subsets must also be frequent. Conversely, if a subset is infrequent, its supersets cannot be frequent.\n\nThe algorithm iteratively generates candidate itemsets of length k from frequent itemsets of length k-1, scans the database to calculate support, and prunes candidates below the threshold. In big data, Apriori can be parallelized using MapReduce or Spark to distribute the candidate generation and support counting phases across multiple nodes."},{"type":"svg","src":"/assets/diagrams/rtu/cs/4/big-data-analytics/diagram-20.svg","alt":"Apriori Algorithm Flow Diagram","caption":"Apriori Algorithm Flow Diagram"}]}},{"id":"v2-big-data-analytics-2022-pC-q4","number":21,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Explain Hive architecture and its data types."}],"answer":{"summary":"Hive provides an SQL-like interface over Hadoop, utilizing components like the Metastore, Driver, and Execution Engine.","explanation":[{"type":"paragraph","content":"Apache Hive allows querying and managing large datasets residing in distributed storage. Its architecture consists of:\n\n1. **Clients**: Drivers like JDBC, ODBC, and Thrift allow different applications to connect to Hive.\n2. **Hive Services**: Includes the CLI, Web UI, and the Hive Server for client connections.\n3. **Driver & Compiler**: The Driver receives the HiveQL query. The Compiler parses the query, performs semantic analysis, and generates an execution plan with the help of the Metastore.\n4. **Metastore**: A central repository that stores the metadata (schema, tables, partitions) of the data stored in HDFS. It typically uses an RDBMS like MySQL.\n5. **Execution Engine**: Executes the plan generated by the compiler in the proper order by converting it into MapReduce, Tez, or Spark jobs.\n\n**Hive Data Types** include Primitive types (TINYINT, INT, BIGINT, FLOAT, DOUBLE, STRING, BOOLEAN, TIMESTAMP) and Complex types (ARRAY, MAP, STRUCT, UNION)."},{"type":"svg","src":"/assets/diagrams/rtu/cs/4/big-data-analytics/diagram-21.svg","alt":"Hive Architecture Diagram","caption":"Hive Architecture Diagram"}]}},{"id":"v2-big-data-analytics-2022-pC-q5","number":22,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Detail the Spark architecture and RDD operations."}],"answer":{"summary":"Spark architecture involves a Driver Program, Cluster Manager, and Worker Nodes operating on RDDs via Transformations and Actions.","explanation":[{"type":"paragraph","content":"Apache Spark is a unified analytics engine for large-scale data processing. Its architecture operates in a master-worker fashion:\n\n1. **Driver Program**: The process that runs the `main()` function of the application and creates the SparkContext. It converts the user's code into tasks and schedules them on executors.\n2. **Cluster Manager**: A pluggable component (like YARN, Mesos, or Spark's Standalone Manager) responsible for acquiring resources on the cluster.\n3. **Worker Nodes**: Nodes that run application code in the cluster. They host **Executors**, which are processes that run tasks and keep data in memory or disk storage.\n\n**RDD (Resilient Distributed Dataset)** operations are divided into two types:\n- **Transformations**: Lazy operations that create a new RDD from an existing one (e.g., `map`, `filter`, `flatMap`). They do not compute their results immediately.\n- **Actions**: Operations that return a value to the driver program or write data to storage, triggering the execution of transformations (e.g., `reduce`, `collect`, `count`)."},{"type":"svg","src":"/assets/diagrams/rtu/cs/4/big-data-analytics/diagram-22.svg","alt":"Spark Architecture Diagram","caption":"Spark Architecture Diagram"}]}}]},{"id":"big-data-analytics-2023","title":"Big Data Analytics (2023)","subjectSlug":"big-data-analytics","subjectName":"Big Data Analytics","universitySlug":"rtu","branchSlug":"computer-science","year":4,"semester":8,"examYear":2023,"paperCode":"8E1809","questions":[{"id":"v2-big-data-analytics-2023-pA-q1","number":1,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"Define the 'Veracity' characteristic of Big Data."}],"answer":{"summary":"Veracity refers to the quality, accuracy, and trustworthiness of the data.","explanation":[{"type":"paragraph","content":"In Big Data, data is collected from multiple diverse sources, leading to inconsistencies, missing values, and noise. Veracity emphasizes the need to clean and validate data to ensure reliable analytics outcomes."}]}},{"id":"v2-big-data-analytics-2023-pA-q2","number":2,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is a Data Lake?"}],"answer":{"summary":"A Data Lake is a centralized repository that stores vast amounts of raw data in its native format.","explanation":[{"type":"paragraph","content":"Unlike a data warehouse which stores structured data ready for analysis, a data lake accepts structured, semi-structured, and unstructured data (like logs, XML, multimedia) without applying a schema until the data is read (schema-on-read)."}]}},{"id":"v2-big-data-analytics-2023-pA-q3","number":3,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is the function of the Secondary NameNode?"}],"answer":{"summary":"The Secondary NameNode merges the HDFS namespace image (FSImage) with the edit logs to prevent the edit logs from becoming too large.","explanation":[{"type":"paragraph","content":"It is NOT a backup for the primary NameNode. Instead, it performs periodic checkpoints to create an updated FSImage, helping the primary NameNode start up faster in case of a restart."}]}},{"id":"v2-big-data-analytics-2023-pA-q4","number":4,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is a DataFrame in Spark?"}],"answer":{"summary":"A DataFrame is a distributed collection of data organized into named columns, similar to a table in a relational database.","explanation":[{"type":"paragraph","content":"Introduced in Spark SQL, DataFrames provide higher-level APIs than RDDs. They allow Spark to leverage its Catalyst optimizer to execute queries much more efficiently than raw RDD operations."}]}},{"id":"v2-big-data-analytics-2023-pA-q5","number":5,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"State the purpose of GraphX."}],"answer":{"summary":"GraphX is Apache Spark's API for graphs and graph-parallel computation.","explanation":[{"type":"paragraph","content":"It extends the Spark RDD by introducing a new Graph abstraction: a directed multigraph with properties attached to each vertex and edge. It is used for implementing graph algorithms like PageRank, connected components, and triangle counting efficiently."}]}},{"id":"v2-big-data-analytics-2023-pA-q6","number":6,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is Zookeeper used for?"}],"answer":{"summary":"Apache ZooKeeper is a centralized service for maintaining configuration information, naming, and providing distributed synchronization.","explanation":[{"type":"paragraph","content":"In the Hadoop ecosystem, it coordinates distributed applications by providing a reliable coordination service. For example, it is used by HBase for cluster state management and by HDFS for High Availability setup."}]}},{"id":"v2-big-data-analytics-2023-pA-q7","number":7,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is Flume?"}],"answer":{"summary":"Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating, and moving large amounts of log data.","explanation":[{"type":"paragraph","content":"It is highly customizable and often used to stream log data from web servers or application servers directly into HDFS or HBase for long-term storage and analysis."}]}},{"id":"v2-big-data-analytics-2023-pA-q8","number":8,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"Define supervised machine learning."}],"answer":{"summary":"Supervised learning is an ML approach where models are trained using labeled datasets.","explanation":[{"type":"paragraph","content":"The algorithm learns the relationship between input features and the known output labels. Once trained, the model can predict the output for new, unseen data. Common examples include classification and regression tasks."}]}},{"id":"v2-big-data-analytics-2023-pA-q9","number":9,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is a Document Store database?"}],"answer":{"summary":"A Document Store is a type of NoSQL database that stores data in JSON, BSON, or XML documents.","explanation":[{"type":"paragraph","content":"Each document contains key-value pairs, but unlike a simple key-value store, the database understands the internal structure of the document, allowing for complex queries and indexing on nested attributes. MongoDB is a prime example."}]}},{"id":"v2-big-data-analytics-2023-pA-q10","number":10,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"Define TF-IDF."}],"answer":{"summary":"TF-IDF (Term Frequency-Inverse Document Frequency) is a numerical statistic used to reflect the importance of a word in a document relative to a corpus.","explanation":[{"type":"paragraph","content":"Term Frequency measures how often a word occurs in a document, while Inverse Document Frequency penalizes common words (like 'the' or 'and') that appear across many documents. It is widely used in text analytics and NLP."}]}},{"id":"v2-big-data-analytics-2023-pB-q1","number":11,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Explain the HDFS read and write process."}],"answer":{"summary":"Reading and writing to HDFS involves interactions between the Client, NameNode, and DataNodes.","explanation":[{"type":"paragraph","content":"For a **Write**: The client asks the NameNode for permission and a list of DataNodes to store the blocks. The client then writes data directly to the first DataNode in a pipeline. The first DataNode replicates it to the second, which replicates to the third. For a **Read**: The client queries the NameNode for the block locations of a file. The NameNode returns a list of DataNodes sorted by their proximity to the client. The client then reads directly from the closest DataNode."}]}},{"id":"v2-big-data-analytics-2023-pB-q2","number":12,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Describe the 5 V's of Big Data."}],"answer":{"summary":"The 5 V's define the core characteristics of big data: Volume, Velocity, Variety, Veracity, and Value.","explanation":[{"type":"paragraph","content":"**Volume** is the sheer scale of data. **Velocity** is the speed at which data is generated and processed. **Variety** refers to the different formats (structured, unstructured, semi-structured). **Veracity** highlights data reliability and quality issues. **Value** represents the actionable insights and business intelligence derived from processing the data."}]}},{"id":"v2-big-data-analytics-2023-pB-q3","number":13,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Walk through a MapReduce word count example."}],"answer":{"summary":"The Word Count program splits text, maps words to a count of 1, and reduces by summing counts for each word.","explanation":[{"type":"paragraph","content":"1. **Input Split**: A text file is split into chunks.\n2. **Map Phase**: Each chunk is processed line by line. For each word found, a key-value pair `(word, 1)` is emitted.\n3. **Shuffle and Sort**: The framework groups all values associated with the same key (word). For example, `('data', [1, 1, 1])`.\n4. **Reduce Phase**: The reducer iterates over the values list, summing them up to output the final count `('data', 3)`."}]}},{"id":"v2-big-data-analytics-2023-pB-q4","number":14,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Distinguish between transformations and actions in Spark."}],"answer":{"summary":"Transformations build a logical execution plan lazily, while actions trigger the actual execution of the plan.","explanation":[{"type":"paragraph","content":"**Transformations** (e.g., `map()`, `filter()`, `join()`) create a new RDD from an existing one but do not compute the result immediately. Spark simply records the lineage. **Actions** (e.g., `count()`, `collect()`, `saveAsTextFile()`) force the computation of all preceding transformations to return a value to the driver or write data to an external storage system."}]}},{"id":"v2-big-data-analytics-2023-pB-q5","number":15,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Explain the HBase architecture."}],"answer":{"summary":"HBase is a distributed, column-oriented database built on top of HDFS, relying on HMaster and RegionServers.","explanation":[{"type":"paragraph","content":"The architecture includes:\n1. **HMaster**: The master server responsible for monitoring RegionServers, handling metadata changes, and assigning regions.\n2. **RegionServers**: The worker nodes that serve data for reading and writing. They manage **Regions** (horizontal partitions of tables).\n3. **ZooKeeper**: Maintains the live cluster state, tracks the active HMaster, and handles RegionServer crashes.\nData is stored in **HFiles** in HDFS, and writes are first buffered in a memory space called **MemStore** and logged in a **Write-Ahead Log (WAL)** for durability."}]}},{"id":"v2-big-data-analytics-2023-pB-q6","number":16,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Describe the PageRank algorithm."}],"answer":{"summary":"PageRank is a link analysis algorithm used to measure the relative importance of web pages within a graph.","explanation":[{"type":"paragraph","content":"Developed by Google, PageRank assumes that a link from Page A to Page B represents a 'vote' of importance for Page B. The rank of a page depends on the number and quality of links pointing to it. In Big Data, this is implemented iteratively using graph-processing frameworks like Spark GraphX, repeatedly updating each vertex's rank based on its neighbors until convergence."}]}},{"id":"v2-big-data-analytics-2023-pB-q7","number":17,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Explain the Machine learning lifecycle using Spark MLlib."}],"answer":{"summary":"The Spark MLlib lifecycle involves data ingestion, feature engineering, model training, evaluation, and deployment via Pipelines.","explanation":[{"type":"paragraph","content":"Spark MLlib standardizes the ML process using the `Pipeline` API. \n1. **Data Preparation**: Loading raw data into Spark DataFrames.\n2. **Feature Engineering**: Using `Transformers` to extract and convert features (e.g., Tokenization, StringIndexing, VectorAssembler).\n3. **Model Training**: Using `Estimators` (algorithms like Random Forest or Logistic Regression) that train on the DataFrame to produce a `Model` (a Transformer).\n4. **Evaluation**: Testing the model's accuracy on validation data.\n5. **Tuning**: Using tools like CrossValidator to optimize hyperparameters."}]}},{"id":"v2-big-data-analytics-2023-pC-q1","number":18,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Explain HDFS architecture and its fault tolerance mechanisms."}],"answer":{"summary":"HDFS ensures high availability and fault tolerance through replication, heartbeats, and secondary NameNodes.","explanation":[{"type":"paragraph","content":"HDFS is designed to reliably store massive datasets on commodity hardware. Its architecture includes the NameNode (metadata manager) and DataNodes (block storage).\n**Fault Tolerance Mechanisms**:\n1. **Data Replication**: Every data block is replicated across multiple DataNodes (default replication factor is 3). If a node fails, the NameNode detects it and initiates replication of the lost blocks from the surviving replicas to other nodes.\n2. **Heartbeats and Block Reports**: DataNodes send periodic heartbeats to the NameNode. If a heartbeat is not received within a specific time, the node is marked dead. Block reports inform the NameNode about the blocks held by a DataNode.\n3. **High Availability (HA)**: In modern Hadoop, an active/standby NameNode setup ensures that if the active NameNode fails, the standby takes over instantly using shared edit logs maintained by JournalNodes."},{"type":"svg","src":"/assets/diagrams/rtu/cs/4/big-data-analytics/diagram-18.svg","alt":"HDFS Architecture and Fault Tolerance Diagram","caption":"HDFS Architecture and Fault Tolerance Diagram"}]}},{"id":"v2-big-data-analytics-2023-pC-q2","number":19,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Detail the complete MapReduce execution flow."}],"answer":{"summary":"The MapReduce execution flow encompasses input splitting, mapping, shuffling/sorting, reducing, and output generation.","explanation":[{"type":"paragraph","content":"The complete flow of a MapReduce job:\n1. **InputFormat**: Validates the input format and splits the input data into logical `InputSplits`. It provides a `RecordReader` to extract key-value pairs.\n2. **Map Phase**: The `RecordReader` passes key-value pairs to the `map()` function. The Mapper processes them and emits intermediate key-value pairs.\n3. **Combiner (Optional)**: Acts as a mini-reducer to aggregate data locally on the mapper node, reducing network traffic.\n4. **Partitioner**: Determines which reducer will process a specific key. It ensures all values for a single key go to the same reducer.\n5. **Shuffle and Sort**: The framework fetches intermediate outputs from mappers over the network, then sorts and merges them by key.\n6. **Reduce Phase**: The `reduce()` function receives the key and an iterator over its values, processes them, and emits final key-value pairs.\n7. **OutputFormat**: Writes the final key-value pairs to HDFS."},{"type":"svg","src":"/assets/diagrams/rtu/cs/4/big-data-analytics/diagram-19.svg","alt":"MapReduce Execution Flow Diagram","caption":"MapReduce Execution Flow Diagram"}]}},{"id":"v2-big-data-analytics-2023-pC-q3","number":20,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Classify NoSQL databases and explain the CAP Theorem in detail."}],"answer":{"summary":"NoSQL databases fall into four categories, each making specific trade-offs according to the CAP Theorem.","explanation":[{"type":"paragraph","content":"**NoSQL Classifications**:\n1. **Key-Value Stores** (e.g., Redis, DynamoDB): Store data as a hash table where a unique key points to an item.\n2. **Document Stores** (e.g., MongoDB, Couchbase): Store semi-structured data like JSON; queries can be made on document attributes.\n3. **Wide-Column Stores** (e.g., Cassandra, HBase): Store data in tables with flexible columns, optimized for fast aggregations.\n4. **Graph Databases** (e.g., Neo4j): Store data as nodes and edges to handle complex relationships.\n\n**The CAP Theorem**:\nProposes that a distributed system can provide at most two of the following:\n- **Consistency (C)**: Every read receives the most recent write or an error.\n- **Availability (A)**: Every request receives a response, without guarantee it contains the most recent write.\n- **Partition Tolerance (P)**: The system continues to operate despite arbitrary network partitions.\nSince network partitions (P) are unavoidable in distributed systems, NoSQL systems typically choose between CP (e.g., HBase, MongoDB) and AP (e.g., Cassandra)."},{"type":"svg","src":"/assets/diagrams/rtu/cs/4/big-data-analytics/diagram-20.svg","alt":"CAP Theorem Triangle Diagram","caption":"CAP Theorem Triangle Diagram"}]}},{"id":"v2-big-data-analytics-2023-pC-q4","number":21,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Describe Apache Mahout and its machine learning capabilities."}],"answer":{"summary":"Apache Mahout is a scalable machine learning library designed to build scalable ML applications on top of Hadoop.","explanation":[{"type":"paragraph","content":"Mahout provides implementations of various machine learning algorithms focused primarily on collaborative filtering, clustering, and classification.\n1. **Collaborative Filtering**: Algorithms to build recommendation engines (e.g., user-based or item-based recommenders).\n2. **Clustering**: Grouping items based on similarity without prior labels. Mahout includes implementations of K-Means, Fuzzy K-Means, and Canopy clustering.\n3. **Classification**: Categorizing unseen data into pre-defined classes. Mahout supports algorithms like Naive Bayes, Random Forests, and Hidden Markov Models.\nInitially, Mahout used MapReduce for distributed processing, but it has largely migrated to Apache Spark, Flink, and H2O to take advantage of their much faster in-memory processing capabilities via the Samsara vector math experimentation environment."},{"type":"svg","src":"/assets/diagrams/rtu/cs/4/big-data-analytics/diagram-21.svg","alt":"Mahout Architecture Diagram","caption":"Mahout Architecture Diagram"}]}},{"id":"v2-big-data-analytics-2023-pC-q5","number":22,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Explain Streaming data processing with Spark Streaming."}],"answer":{"summary":"Spark Streaming enables scalable, high-throughput, fault-tolerant processing of live data streams using micro-batching.","explanation":[{"type":"paragraph","content":"Instead of processing data record-by-record, Spark Streaming uses a micro-batch architecture:\n1. **Data Ingestion**: It ingests live data streams from sources like Kafka, Flume, or TCP sockets.\n2. **Micro-Batching**: The continuous stream of data is divided into small batches of a specific time interval (e.g., 1 second).\n3. **DStream (Discretized Stream)**: The abstraction used by Spark Streaming, representing a continuous sequence of RDDs. Each RDD contains the data received during a specific time interval.\n4. **Processing**: The Spark engine processes these RDDs using complex algorithms expressed with high-level functions like `map`, `reduce`, `join`, and `window`.\n5. **Output**: The processed batches can be pushed to databases, HDFS, or live dashboards.\nBecause it relies on Spark Core's RDDs, it inherits strong fault-tolerance, automatically recovering lost state and data upon node failure."},{"type":"svg","src":"/assets/diagrams/rtu/cs/4/big-data-analytics/diagram-22.svg","alt":"Spark Streaming Flow Diagram","caption":"Spark Streaming Flow Diagram"}]}}]},{"id":"big-data-analytics-2026","title":"Big Data Analytics (2026)","subjectSlug":"big-data-analytics","subjectName":"Big Data Analytics","universitySlug":"rtu","branchSlug":"computer-science","year":4,"semester":8,"examYear":2026,"paperCode":"8E1809","questions":[{"id":"v2-big-data-analytics-2026-pA-q1","number":1,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is Data Science?"}],"answer":{"summary":"Data Science is the highly multidisciplinary mathematical field combining statistics, advanced algorithms, and machine learning architectures to violently extract hidden, actionable intelligence from massive datasets.","explanation":[{"type":"paragraph","content":"It completely abandons primitive SQL querying. Data Science mathematically predicts future events (like stock market crashes or cancer cell mutation) by training complex neural networks and regression models on massive historical data, bridging the gap between computer science and pure probability physics."}]}},{"id":"v2-big-data-analytics-2026-pA-q2","number":2,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"Define unstructured data."}],"answer":{"summary":"Unstructured data is chaotic, raw information completely lacking a predefined mathematical data model or rigid schema, rendering it unreadable to standard relational SQL databases.","explanation":[{"type":"paragraph","content":"Roughly 80% of global data is unstructured. It includes massive raw video files, 10,000-page PDF documents, chaotic social media text, and audio recordings. Because it lacks rows and columns, it requires catastrophic processing power (like Hadoop or NLP algorithms) to mathematically parse and understand."}]}},{"id":"v2-big-data-analytics-2026-pA-q3","number":3,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What was the role of the JobTracker in Hadoop 1.x?"}],"answer":{"summary":"In the primitive Hadoop 1.x architecture, the JobTracker was the absolute centralized master node mathematically responsible for both cluster resource management and MapReduce job scheduling.","explanation":[{"type":"paragraph","content":"The JobTracker was a catastrophic Single Point of Failure and a massive architectural bottleneck. It had to violently track the exact status of 10,000 tasks across the entire cluster while simultaneously managing CPU/RAM resources. If it crashed, the entire massive cluster instantly went dead."}]}},{"id":"v2-big-data-analytics-2026-pA-q4","number":4,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is Apache Oozie?"}],"answer":{"summary":"Apache Oozie is a massive, server-based workflow scheduling architecture mathematically designed strictly to manage and orchestrate complex, dependent Hadoop MapReduce and Pig/Hive jobs.","explanation":[{"type":"paragraph","content":"In a massive Big Data pipeline, Job B cannot mathematically execute until Job A successfully finishes. Oozie violently manages this dependency using Directed Acyclic Graphs (DAGs), ensuring that if a Hadoop job fails, the downstream processes are halted and the engineer is instantly alerted."}]}},{"id":"v2-big-data-analytics-2026-pA-q5","number":5,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"Give a use case for Spark SQL."}],"answer":{"summary":"Spark SQL is utilized to execute mathematically massive, real-time interactive queries over Petabytes of JSON data stored in an HDFS Data Lake using standard SQL syntax.","explanation":[{"type":"paragraph","content":"A data scientist needs to violently calculate the average transaction value of 100 million credit card swipes from yesterday. Instead of writing 500 lines of complex Java MapReduce code, they simply write `SELECT AVG(value) FROM transactions` and Spark SQL mathematically executes it in memory in seconds."}]}},{"id":"v2-big-data-analytics-2026-pA-q6","number":6,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What are column-family databases?"}],"answer":{"summary":"Column-family databases (like Apache Cassandra or HBase) are massive NoSQL architectures that physically store data strictly by columns rather than traditional SQL rows.","explanation":[{"type":"paragraph","content":"This architectural physics allows for explosive analytical speed. If an engineer wants to calculate the mathematical average of a \"Salary\" column across 1 billion users, the disk head only reads that exact physical column on the hard drive, completely bypassing the massive overhead of reading the entire row."}]}},{"id":"v2-big-data-analytics-2026-pA-q7","number":7,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is Apache Sqoop?"}],"answer":{"summary":"Apache Sqoop is a highly aggressive command-line architecture mathematically engineered strictly to violently transfer massive volumes of data between relational databases (MySQL) and Hadoop (HDFS).","explanation":[{"type":"paragraph","content":"Sqoop completely abstracts the catastrophic complexity of database migration. It uses MapReduce mathematics under the hood to violently slice a 1-Terabyte Oracle SQL table into chunks, opening 50 parallel connections to the database to blast the data directly into HDFS in minutes."}]}},{"id":"v2-big-data-analytics-2026-pA-q8","number":8,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"Define regression in data analytics."}],"answer":{"summary":"Regression is a predictive mathematical algorithm in machine learning that models the strict statistical relationship between independent variables (features) and a continuous dependent variable (target).","explanation":[{"type":"paragraph","content":"Unlike Classification (which predicts categories like \"Spam\" or \"Not Spam\"), Regression predicts an absolute numerical physics value. For example, a Linear Regression model mathematically calculates the exact future price of a house based strictly on its square footage, age, and zip code."}]}},{"id":"v2-big-data-analytics-2026-pA-q9","number":9,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"Name a popular graph database."}],"answer":{"summary":"Neo4j is the absolute industry-standard open-source Graph Database, mathematically engineered to violently map complex, highly connected data topologies.","explanation":[{"type":"paragraph","content":"Unlike SQL databases which struggle catastrophically with deep relational JOINs, Neo4j stores data as Nodes (users) and mathematical Edges (relationships). It can execute a query like \"Find all friends of friends of John who like Python\" in milliseconds, making it the architecture behind massive social networks."}]}},{"id":"v2-big-data-analytics-2026-pA-q10","number":10,"marks":2,"tags":["Part A"],"text":[{"type":"paragraph","content":"What is a Resilient Distributed Dataset (RDD)?"}],"answer":{"summary":"An RDD is the absolute fundamental mathematical data structure of Apache Spark: an immutable, partitioned collection of records distributed across massive server clusters.","explanation":[{"type":"paragraph","content":"Resilient Distributed Datasets violently solve the catastrophic disk I/O bottleneck of Hadoop. An RDD forces the 1-Terabyte dataset to reside entirely in physical RAM across 100 servers. It tracks its own mathematical lineage, meaning if a server burns down, Spark can instantly recompute the lost RAM partition from scratch."}]}},{"id":"v2-big-data-analytics-2026-pB-q1","number":11,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Describe a typical Data processing pipeline in Big Data."}],"answer":{"summary":"An architectural breakdown of the Big Data Processing Pipeline. Violently details the strict physical sequence of Data Ingestion (Kafka), Storage (HDFS), Processing (Spark), and Visualization, exposing how massive data flows from edge to insight.","explanation":[{"type":"heading","content":"The Architecture of a Big Data Processing Pipeline"},{"type":"paragraph","content":"A Big Data Pipeline is not a single software application; it is a massive, highly complex algorithmic architecture that violently transports, scrubs, and analyzes Petabytes of chaotic data. If any physical stage of this pipeline bottlenecks, the entire analytics system mathematically collapses."},{"type":"subheading","content":"1. Data Ingestion (The Funnel)"},{"type":"paragraph","content":"The absolute front line. The system must violently ingest data from 10,000 completely different sources (IoT sensors, SQL databases, Twitter APIs) in real-time. Tools like Apache Kafka or Flume mathematically buffer this catastrophic tsunami of data, guaranteeing zero packet loss even if the downstream servers are overwhelmed."},{"type":"subheading","content":"2. Data Storage (The Data Lake)"},{"type":"paragraph","content":"The ingested data is raw, dirty, and chaotic. It cannot be stored in a rigid SQL database. It is violently dumped exactly as-is into a massive Data Lake (Hadoop HDFS or AWS S3). The architecture mathematically splits these massive files across thousands of cheap commodity hard drives using replication to guarantee fault tolerance."},{"type":"subheading","content":"3. Data Processing and Transformation (ETL)"},{"type":"paragraph","content":"The raw data is mathematically useless. The pipeline executes Extract, Transform, Load (ETL). Massive distributed engines like Apache Spark or Hadoop MapReduce violently crush the data. They scrub out null values, mathematically normalize timestamps, and structure the data into highly optimized formats like Apache Parquet."},{"type":"subheading","content":"4. Data Analytics and Machine Learning"},{"type":"paragraph","content":"The clean data is now violently assaulted by mathematical algorithms. Data Scientists train massive Neural Networks or execute complex Hive SQL queries against the dataset to extract predictive intelligence (e.g., predicting exactly which server will crash next week)."},{"type":"subheading","content":"5. Data Visualization (The Output)"},{"type":"paragraph","content":"The final mathematical results are pushed to an enterprise dashboard (Tableau, Grafana). The CEO does not read raw JSON; they look at a live, mathematically rendered heatmap dictating corporate strategy."}]}},{"id":"v2-big-data-analytics-2026-pB-q2","number":12,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"What are the key differences between Hadoop 1.x and Hadoop 2.x?"}],"answer":{"summary":"A critical architectural comparison between Hadoop 1.x and Hadoop 2.x. Details the catastrophic bottleneck of the monolithic JobTracker and how the invention of YARN violently decentralized cluster resource management, enabling real-time processing beyond MapReduce.","explanation":[{"type":"heading","content":"Hadoop 1.x vs. Hadoop 2.x: The Architectural Evolution"},{"type":"paragraph","content":"Hadoop 1.x was a pioneering architecture, but it contained a catastrophic mathematical flaw that strictly limited its scaling to 4,000 nodes. Hadoop 2.x completely ripped out the core resource management engine and replaced it with a massively decentralized architecture called YARN (Yet Another Resource Negotiator)."},{"type":"heading","content":"The Catastrophic Flaw in Hadoop 1.x"},{"type":"list","items":["**Monolithic JobTracker:** In 1.x, exactly ONE massive server (the JobTracker) was mathematically responsible for EVERYTHING. It had to schedule 100,000 MapReduce tasks AND track the exact CPU/RAM usage of 4,000 worker nodes. It violently bottlenecked the entire cluster.","**Single Point of Failure (SPOF):** If the JobTracker hardware caught fire, the ENTIRE multi-million dollar cluster instantly died. There was zero High Availability.","**MapReduce Only:** The cluster was mathematically hardcoded to ONLY execute batch MapReduce jobs. You could not run real-time streaming (like Spark) or graph processing on the same servers."]},{"type":"heading","content":"The YARN Revolution in Hadoop 2.x"},{"type":"paragraph","content":"Hadoop 2.x violently separated Resource Management from Job Scheduling by creating YARN."},{"type":"list","items":["**ResourceManager & ApplicationMaster:** The central ResourceManager now ONLY handles CPU/RAM allocation. When a user submits a job, a dedicated, temporary ApplicationMaster is violently booted up just for that specific job to handle task tracking. This mathematically removes the bottleneck, allowing the cluster to scale past 10,000 nodes.","**High Availability (HA):** Hadoop 2.x introduced multiple NameNodes. If the active master server dies, Zookeeper mathematically detects it in milliseconds and violently fails over to a Standby Master with zero data loss.","**Multi-Paradigm Processing:** YARN acts as a Cloud Operating System. The exact same Hadoop cluster can now simultaneously run MapReduce (batch), Apache Spark (in-memory), and Apache Storm (real-time streaming), violently maximizing hardware ROI."]}]}},{"id":"v2-big-data-analytics-2026-pB-q3","number":13,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"How does Sqoop work?"}],"answer":{"summary":"A definitive explanation of Apache Sqoop's underlying architecture. Details how it completely bypasses traditional slow ETL by mathematically compiling Java code and launching massive parallel Map-Only jobs to violently extract data from SQL servers into HDFS.","explanation":[{"type":"heading","content":"The Internal Architecture and Mechanism of Apache Sqoop"},{"type":"paragraph","content":"Apache Sqoop (SQL-to-Hadoop) is not a simple script; it is a highly aggressive, massively parallel data transfer architecture. When a corporation needs to move 5 Terabytes of transaction data from a rigid Oracle database into HDFS for Big Data analytics, doing it sequentially (row by row) would mathematically take days. Sqoop violently executes this in minutes."},{"type":"heading","content":"The Mathematical Execution Pipeline"},{"type":"list","items":["**Step 1: The Metadata Inspection:** When the user runs a Sqoop `import` command, Sqoop does NOT immediately touch the data. It connects to the RDBMS (Oracle/MySQL) and mathematically queries the schema metadata. It finds out exactly what columns exist and what their physical data types are (INT, VARCHAR).","**Step 2: Dynamic Java Compilation:** Sqoop takes this metadata and violently auto-generates custom Java classes on the fly. It compiles this code, which will be used to mathematically serialize the SQL rows into Hadoop-friendly objects.","**Step 3: Bounding Box Calculation (The Split):** Sqoop queries the Primary Key of the table. If the IDs range from 1 to 10,000,000, and the user requested 4 parallel tasks (`-m 4`), Sqoop mathematically slices the table into four exact chunks (1-2.5M, 2.5M-5M, etc.).","**Step 4: The Map-Only Execution:** This is the absolute genius of the architecture. Sqoop submits a strict **Map-Only MapReduce Job** to the Hadoop cluster (there is NO Reduce phase, as no aggregation is needed).","**Step 5: Parallel Data Blast:** Four separate Hadoop worker nodes simultaneously open four separate JDBC network connections to the Oracle database. Node A executes `SELECT * WHERE id < 2500000`. Node B handles the next chunk. They violently suck the data out in parallel and dump it directly into distributed HDFS blocks."]}]}},{"id":"v2-big-data-analytics-2026-pB-q4","number":14,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Explain Hive partitioning and bucketing."}],"answer":{"summary":"A critical mathematical breakdown of Apache Hive performance tuning. Details how Partitioning physically isolates data into rigid directory structures, while Bucketing uses modulo hash functions to violently cluster data for explosive JOIN performance.","explanation":[{"type":"heading","content":"Data Organization Architecture in Apache Hive"},{"type":"paragraph","content":"Apache Hive allows users to query Petabytes of HDFS data using SQL. However, if a user runs `SELECT * FROM logs WHERE date=\"2023-10-01\"`, Hive will mathematically trigger a \"Full Table Scan,\" forcing thousands of servers to violently read every single Petabyte of data on the cluster just to find one day's worth of logs. This is a catastrophic architectural failure. To solve this, Hive implements Partitioning and Bucketing."},{"type":"heading","content":"1. Hive Partitioning (Macro-Isolation)"},{"type":"paragraph","content":"Partitioning physically alters the HDFS directory structure based on the strict value of a column."},{"type":"list","items":["**Mechanism:** If we partition the `logs` table by `year` and `month`, Hive mathematically creates physical folders on the hard drives: `/logs/year=2023/month=10/`.","**The Merit (Partition Pruning):** When the query runs for October 2023, the Hive compiler mathematically realizes it ONLY needs to read the exact physical folder `/month=10/`. It violently ignores 99% of the other folders, reducing a 5-hour query to 5 seconds.","**Demerit:** If you partition on a column with a million unique values (like `User_ID`), it creates a million tiny HDFS folders, violently crashing the Hadoop NameNode (The Small File Problem)."]},{"type":"heading","content":"2. Hive Bucketing (Micro-Clustering via Hashing)"},{"type":"paragraph","content":"Bucketing solves the Small File Problem and violently optimizes massive SQL `JOIN` operations."},{"type":"list","items":["**Mechanism:** The user explicitly forces the table into a strict number of buckets (e.g., 32 files). Hive mathematically takes the `User_ID`, executes a Hash Function on it, and calculates `Hash(User_ID) % 32`.","**The Execution:** The user's data is physically dumped into that specific bucket file (0 to 31).","**The Merit (Map-Side Joins):** If you join two massively bucketed tables on `User_ID`, the Hadoop engine knows mathematically that `User_ID=500` in Table A and `User_ID=500` in Table B are BOTH strictly residing in Bucket 4. It can join them locally in RAM without executing a catastrophic network shuffle."]}]}},{"id":"v2-big-data-analytics-2026-pB-q5","number":15,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Compare Spark processing with Hadoop MapReduce."}],"answer":{"summary":"A definitive architectural contrast between Apache Spark and Hadoop MapReduce. Violently details the catastrophic disk I/O bottleneck of MapReduce against Spark's explosive in-memory RDD architecture and Directed Acyclic Graph (DAG) execution.","explanation":[{"type":"heading","content":"Architectural Comparison: Apache Spark vs. Hadoop MapReduce"},{"type":"paragraph","content":"Hadoop MapReduce invented the Big Data industry, but it is mathematically obsolete for modern iterative processing. Apache Spark was engineered from the ground up to completely annihilate the physical Disk I/O bottlenecks that crippled MapReduce, resulting in processing speeds up to 100x faster."},{"type":"heading","content":"1. The Physical Memory Architecture (Disk vs. RAM)"},{"type":"list","items":["**MapReduce (Disk-Bound):** MapReduce is mathematically paranoid about hardware failure. After the Map phase finishes, it violently writes every single byte of intermediate data back to the physical Hard Drive. The Reduce phase then reads it back from the Hard Drive. If an algorithm requires 10 iterations (like Machine Learning), it executes 10 catastrophic write/read cycles to slow, spinning disks.","**Spark (In-Memory):** Spark utilizes Resilient Distributed Datasets (RDDs). It mathematically forces the intermediate data to stay locked entirely in physical RAM across the cluster. It never touches the hard drive unless explicitly commanded. This completely destroys the latency bottleneck."]},{"type":"heading","content":"2. The Execution Engine (Sequential vs. DAG)"},{"type":"list","items":["**MapReduce (Rigid Sequence):** It is forced into a strict two-step box: Map, then Reduce. You cannot mathematically change this pipeline.","**Spark (DAG Scheduler):** Spark builds a highly complex Directed Acyclic Graph (DAG) of operations before it executes anything. It mathematically analyzes the entire pipeline, optimizes the query, and pipelines multiple operations (Map, Filter, Reduce) into a single, seamless, memory-bound execution stage."]},{"type":"heading","content":"3. Data Processing Paradigms"},{"type":"list","items":["**MapReduce:** Strictly handles Batch processing. It physically cannot handle real-time streaming data.","**Spark:** The absolute unified engine. The exact same Spark cluster can execute massive SQL queries, train Machine Learning models (MLlib), analyze Graph architectures (GraphX), and process sub-second real-time streams (Spark Streaming) using the exact same underlying RAM architecture."]}]}},{"id":"v2-big-data-analytics-2026-pB-q6","number":16,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"Explain the concept of Decision trees in big data ML."}],"answer":{"summary":"A rigorous mathematical exposition of Decision Trees in Machine Learning. Details how the algorithm recursively partitions massive datasets using Information Gain and Entropy mathematics to construct a rigid, interpretable predictive architecture.","explanation":[{"type":"heading","content":"Decision Tree Architecture in Big Data Machine Learning"},{"type":"paragraph","content":"A Decision Tree is a highly interpretable, non-parametric Supervised Machine Learning algorithm used for both Classification (predicting categories) and Regression (predicting numbers). It completely abandons the \"black-box\" nature of Neural Networks, instead creating a massive mathematical flowchart that human engineers can physically read and audit."},{"type":"heading","content":"The Mathematical Construction (Recursive Partitioning)"},{"type":"paragraph","content":"The algorithm does not guess; it uses brutal statistical mathematics to build the tree."},{"type":"list","items":["**The Root Node:** The algorithm analyzes a dataset of 1 million customers. It must find the absolute best feature (e.g., \"Age\" or \"Income\") to split the data into two highly distinct groups (e.g., \"Will Default on Loan\" vs \"Will Pay\").","**The Math (Entropy & Information Gain):** It mathematically calculates the Entropy (chaos/impurity) of the dataset. It then tests every single variable to see which split causes the most massive drop in Entropy. This maximum drop is called the Information Gain.","**The Splitting:** If \"Income < $50,000\" yields the highest Information Gain, that becomes the Root Node. The data is violently sliced in half.","**Recursion:** The algorithm recursively repeats this exact mathematical process on the new branches, creating internal nodes (further questions) until it reaches a Leaf Node (the final absolute prediction)."]},{"type":"heading","content":"Merits and Catastrophic Demerits in Big Data"},{"type":"list","items":["**Merits:** Zero requirement for data normalization. It mathematically handles missing values and nonlinear relationships effortlessly. Absolute human interpretability for legal/compliance reasons.","**Demerits (Overfitting):** A Decision Tree will aggressively keep growing until it perfectly memorizes every single row in the training data, capturing random noise instead of real patterns. In production, this overfitted tree will fail catastrophically. It MUST be mathematically controlled using \"Pruning\" (forcing a maximum depth limit) or by combining thousands of trees into a Random Forest architecture."]}]}},{"id":"v2-big-data-analytics-2026-pB-q7","number":17,"marks":4,"tags":["Part B"],"text":[{"type":"paragraph","content":"What are Recommender systems?"}],"answer":{"summary":"A critical review of Recommender Systems. Explains how Netflix and Amazon utilize massive Big Data architectures to mathematically predict user preferences through Content-Based filtering and the highly complex matrix factorization of Collaborative Filtering.","explanation":[{"type":"heading","content":"Architecture of Big Data Recommender Systems"},{"type":"paragraph","content":"Recommender Systems are the absolute financial engine of the modern internet. Amazon, Netflix, and YouTube do not wait for users to search; they use massive Big Data Machine Learning algorithms to mathematically predict exactly what the user wants to consume next, driving billions of dollars in revenue."},{"type":"heading","content":"The Two Core Mathematical Paradigms"},{"type":"subheading","content":"1. Content-Based Filtering"},{"type":"paragraph","content":"This algorithm mathematically analyzes the strict physical attributes (metadata) of the items the user has previously interacted with."},{"type":"list","items":["**Mechanism:** If User A watches 5 movies that are tagged with the attributes `[\"Action\", \"Tom Cruise\", \"Explosions\"]`, the algorithm mathematically creates a User Profile vector. It then scans a database of 10,000 unviewed movies and calculates the Cosine Similarity between the User Vector and the Movie Vectors, recommending the closest match.","**Demerit:** The \"Filter Bubble.\" It will violently ONLY recommend Action movies. It physically cannot discover that the user might also like a Romantic Comedy."]},{"type":"subheading","content":"2. Collaborative Filtering"},{"type":"paragraph","content":"This algorithm completely ignores the metadata of the movie. It strictly analyzes massive matrices of User-Item interactions (ratings, clicks)."},{"type":"list","items":["**Mechanism:** If User A likes Movies 1, 2, and 3. And User B likes Movies 1, 2, and 3. The algorithm mathematically determines User A and User B are a \"Neighborhood.\" If User B suddenly watches and likes Movie 4, the algorithm violently forces Movie 4 into User A's recommendation feed, even if they have completely different metadata.","**Demerit:** The \"Cold Start\" problem. If a brand new movie is uploaded, it has exactly zero user ratings. The Collaborative algorithm is mathematically blind to it and cannot recommend it to anyone until someone randomly clicks it."]}]}},{"id":"v2-big-data-analytics-2026-pC-q1","number":18,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Discuss the complete Big Data Analytics lifecycle."}],"answer":{"summary":"A massive architectural breakdown of the Big Data Analytics Lifecycle. Violently dissects the strict 6-phase algorithmic process from initial Business Discovery to mathematical Model Building, Operationalization, and final strategic delivery.","explanation":[{"type":"heading","content":"The Complete Big Data Analytics Lifecycle Architecture"},{"type":"paragraph","content":"Executing a massive Big Data project is not simply writing Python code and importing a dataset. It is a highly rigid, 6-phase mathematical methodology. If a corporation attempts to skip Phase 2 and jump straight to Phase 4 (Machine Learning), the model will mathematically collapse because it will be trained on garbage data. The lifecycle dictates a strict architectural flow."},{"type":"heading","content":"Phase 1: Discovery (The Business Architecture)"},{"type":"paragraph","content":"Before a single byte of data is touched, the team must mathematically define the business problem."},{"type":"list","items":["**Objective:** What is the exact financial goal? (e.g., \"Reduce customer churn by 15% in Q3\").","**Resources:** Do we have the massive Hadoop clusters required? Do we legally have the right to process this data (GDPR)?","**Hypothesis Generation:** The Data Scientists formulate rigid mathematical hypotheses (e.g., \"Customers who call support 3 times in a week have an 80% probability of churning\")."]},{"type":"heading","content":"Phase 2: Data Preparation (The Analytic Sandbox)"},{"type":"paragraph","content":"The most computationally violent and time-consuming phase (often 80% of the project time)."},{"type":"list","items":["**The Sandbox:** The IT department provisions a massive Hadoop workspace where data scientists can run chaotic queries without crashing the production database.","**ETL (Extract, Transform, Load):** The team violently ingests Petabytes of raw data from 10 different systems. They mathematically scrub the data (handling missing NULL values, fixing broken timestamps, and normalizing massive text fields). If the data is garbage, the AI will output garbage."]},{"type":"heading","content":"Phase 3: Model Planning (The Algorithm Selection)"},{"type":"paragraph","content":"The data is clean. Now the team must architect the mathematical approach."},{"type":"list","items":["**Feature Engineering:** The absolute most critical step. The scientists mathematically combine variables to create new predictive features (e.g., combining `Total_Spend` and `Account_Age` to create a `Loyalty_Ratio`).","**Algorithm Selection:** If the goal is classification, do we use a Random Forest, Logistic Regression, or a massive Deep Neural Network? The scientists select the algorithm that perfectly matches the mathematical shape of the data."]},{"type":"heading","content":"Phase 4: Model Building (Execution)"},{"type":"paragraph","content":"The actual heavy lifting. The data scientists write the Spark MLlib or TensorFlow code."},{"type":"list","items":["**Training:** The massive dataset is violently split (80% training, 20% testing). The algorithm churns through the 80% for hours, mathematically adjusting its internal weights to minimize the error rate.","**Validation:** The model is forced to predict the outcomes of the 20% unseen data. If the model achieves 95% accuracy on the test set, it is mathematically proven to work."]},{"type":"heading","content":"Phase 5: Communicate Results (The Translation)"},{"type":"paragraph","content":"The Data Scientist must translate complex calculus back into business English."},{"type":"list","items":["They present the findings to the CEO using massive Tableau dashboards. They must prove mathematically that the model achieved the financial goal defined in Phase 1."]},{"type":"heading","content":"Phase 6: Operationalize (The Production Deployment)"},{"type":"paragraph","content":"The model is taken out of the sandbox and violently deployed into the live corporate architecture."},{"type":"list","items":["It is wrapped in a REST API. Now, when a live customer calls support for the 3rd time, the live database pings the ML model in milliseconds. The model returns a \"High Churn Risk\" flag, and the system autonomously commands a human agent to offer the customer a 20% discount."]}]}},{"id":"v2-big-data-analytics-2026-pC-q2","number":19,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Explain the Lambda architecture for big data systems."}],"answer":{"summary":"An exhaustive mathematical dissection of the Lambda Architecture. Details how Nathan Marz violently solved the conflict between slow batch processing (Hadoop) and inaccurate real-time streaming (Storm) by fusing them into an impenetrable, immutable three-layer system.","explanation":[{"type":"heading","content":"The Lambda Architecture for Big Data Systems"},{"type":"paragraph","content":"Before the Lambda Architecture, Big Data systems faced a catastrophic paradox. If you used Hadoop MapReduce, you got absolute mathematical accuracy, but the report took 8 hours to generate (useless for real-time fraud detection). If you used Apache Storm (streaming), you got sub-second speed, but the data was often inaccurate due to dropped packets. Nathan Marz invented the **Lambda Architecture** to violently fuse these two paradigms together, guaranteeing both absolute accuracy AND microsecond latency."},{"type":"heading","content":"The Core Principle: Immutable Data"},{"type":"paragraph","content":"Lambda operates on one rigid physical law: Data is completely immutable. Once a record is written, it can NEVER be updated or deleted. If a user changes their address, you do not `UPDATE` the database; you strictly append a new record with a new timestamp. This completely destroys database lock contention and race conditions."},{"type":"heading","content":"The Three Massive Architectural Layers"},{"type":"subheading","content":"1. The Batch Layer (The Source of Truth)"},{"type":"paragraph","content":"This layer uses Hadoop HDFS and MapReduce/Spark."},{"type":"list","items":["**Function:** It stores the absolute master dataset. Every single byte of data ever generated is dumped here.","**Execution:** Every few hours, it violently processes the ENTIRE dataset from the beginning of time to mathematically compute massive \"Batch Views\" (pre-calculated answers).","**Merit:** Absolute mathematical perfection. It resolves all late-arriving or out-of-order data. **Demerit:** It is hours out of date."]},{"type":"subheading","content":"2. The Speed Layer (The Real-Time Engine)"},{"type":"paragraph","content":"This layer uses Apache Kafka and Spark Streaming or Storm."},{"type":"list","items":["**Function:** It only looks at the exact data that has arrived *since the last Batch Layer run began*.","**Execution:** It violently processes this tiny slice of data in milliseconds, generating \"Real-time Views\".","**Merit:** Instantaneous analytics. **Demerit:** It is mathematically approximate. It might drop a packet or double-count a record due to network chaos."]},{"type":"subheading","content":"3. The Serving Layer (The Fusion Interface)"},{"type":"paragraph","content":"This layer uses ultra-fast NoSQL databases like Apache Cassandra or HBase."},{"type":"list","items":["**Function:** It loads the massive Batch Views and the tiny Real-time Views into RAM.","**Execution:** When the user queries the dashboard, the Serving Layer mathematically merges the perfect (but old) Batch View with the chaotic (but new) Real-time View.","**The Result:** The user receives a query response in 10 milliseconds that is mathematically perfectly accurate up to a few hours ago, and highly accurate up to the exact current millisecond."]},{"type":"svg","content":""}]}},{"id":"v2-big-data-analytics-2026-pC-q3","number":20,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Detail real-time analytics using Kafka and Spark."}],"answer":{"summary":"A definitive architectural guide to Real-Time Analytics using Apache Kafka and Spark Streaming. Explains how Kafka's massive distributed log buffers the chaotic data tsunami, and how Spark's Micro-Batching mathematics violently processes the stream in sub-second intervals.","explanation":[{"type":"heading","content":"Real-Time Analytics Architecture: Kafka and Spark Streaming"},{"type":"paragraph","content":"In the modern data architecture, processing data overnight is obsolete. If a credit card is stolen, the fraud must be mathematically detected in 50 milliseconds, not tomorrow morning. This requires a violently robust Real-Time Pipeline. The absolute industry standard for this architecture is the combination of **Apache Kafka** (the shock absorber) and **Apache Spark Streaming** (the mathematical engine)."},{"type":"heading","content":"1. The Shock Absorber: Apache Kafka"},{"type":"paragraph","content":"If 10 million IoT devices suddenly spike and blast data simultaneously, they will mathematically crash any database. Apache Kafka sits at the front door to prevent this."},{"type":"list","items":["**Architecture:** Kafka is a massively distributed, append-only Commit Log. It does not process data; it just violently absorbs it and writes it directly to disk sequentially, allowing it to ingest millions of messages per second with zero lag.","**Topics and Partitions:** The data is mathematically categorized into \"Topics\". Each topic is physically chopped into \"Partitions\" spread across multiple servers. This guarantees catastrophic fault tolerance and allows hundreds of consumers to read the data completely in parallel.","**Decoupling:** Kafka completely decouples the producers (IoT devices) from the consumers (Spark). If Spark crashes and burns for 10 minutes, Kafka safely holds the data. When Spark reboots, it mathematically resumes reading from its exact last offset."]},{"type":"heading","content":"2. The Processing Engine: Spark Streaming"},{"type":"paragraph","content":"Spark reads the chaotic, infinite stream of bytes from Kafka and executes complex mathematical analytics."},{"type":"list","items":["**Micro-Batch Architecture (DStreams):** Traditional Spark operates on massive, static RDDs. Spark Streaming violently chops the infinite Kafka stream into tiny chunks based on time (e.g., exactly 0.5 seconds of data). This tiny chunk is converted into an RDD, and the standard Spark engine crushes it.","**Structured Streaming:** The modern evolution. It mathematically treats the infinite Kafka stream as an infinitely growing SQL table. The developer writes a standard SQL query (`SELECT count(*) FROM stream GROUP BY ip_address`), and Spark mathematically continuously updates the answer in RAM every millisecond.","**Windowing Operations:** Spark can execute complex mathematical analysis over time. E.g., \"Calculate the average temperature over a sliding 10-minute window, updating every 2 seconds.\" It manages the chaotic out-of-order arrival of data using \"Event-Time Watermarking\"."]},{"type":"heading","content":"The Execution Flow"},{"type":"paragraph","content":"1. A credit card is swiped at a terminal. \n2. The terminal blasts the JSON payload into a Kafka Topic. \n3. Spark Streaming instantly pulls the JSON, parses it, and feeds it into a pre-trained Machine Learning Random Forest model residing in RAM. \n4. The model mathematically predicts a 99% probability of fraud. \n5. Spark instantly fires a command back to the banking API to physically decline the transaction, executing the entire architectural loop in under 50 milliseconds."}]}},{"id":"v2-big-data-analytics-2026-pC-q4","number":21,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Describe the Pig execution environment and its main components."}],"answer":{"summary":"An exhaustive theoretical breakdown of Apache Pig. Details how the Pig Latin dataflow language completely abstracts the horrific complexity of raw Java MapReduce, aggressively translating high-level relational operations into optimized mathematical DAGs.","explanation":[{"type":"heading","content":"Apache Pig Architecture and Execution Environment"},{"type":"paragraph","content":"Writing a raw Java MapReduce program to execute a simple `JOIN` between two massive 5-Terabyte datasets requires roughly 300 lines of highly complex, error-prone Java code. It is an architectural nightmare. Yahoo invented **Apache Pig** to violently destroy this complexity. Pig is a high-level Dataflow programming platform that allows analysts to write 5 lines of simple script, which the Pig architecture mathematically compiles into the 300 lines of optimized Java MapReduce."},{"type":"heading","content":"1. Pig Latin (The Language)"},{"type":"paragraph","content":"Pig Latin is not SQL. SQL is a declarative language (you tell the system *what* you want, and the system figures out *how* to do it). Pig Latin is a **Dataflow language**. The programmer mathematically dictates the exact step-by-step physical transformation of the data."},{"type":"list","items":["`A = LOAD \"data.txt\" AS (id:int, name:chararray);` (Ingest data from HDFS)","`B = FILTER A BY id > 100;` (Violently strip out useless records)","`C = GROUP B BY name;` (Mathematically group the data, triggering a massive network shuffle)","`D = FOREACH C GENERATE group, COUNT(B);` (Calculate the aggregation)","`STORE D INTO \"output\";` (Trigger the physical MapReduce execution)"]},{"type":"heading","content":"2. The Pig Execution Environment (The Compiler Architecture)"},{"type":"paragraph","content":"When a user submits a Pig script, it is not executed immediately. It passes through a rigid mathematical compilation pipeline:"},{"type":"list","items":["**Parser:** Validates the absolute syntax of the Pig Latin script and verifies the data types. It generates a Directed Acyclic Graph (DAG) representing the logical dataflow.","**Optimizer:** The absolute genius of Pig. The Logical Optimizer mathematically inspects the DAG. If the user wrote a `JOIN` followed by a `FILTER`, the optimizer violently rearranges the code, pushing the `FILTER` *before* the `JOIN`. This mathematically drops 90% of the data before the network shuffle, reducing processing time from hours to minutes.","**Compiler:** It takes the optimized logical DAG and mathematically translates it into a physical sequence of actual MapReduce jobs (e.g., Job 1: Map-only. Job 2: Map and Reduce).","**Execution Engine:** It blindly submits the compiled `.jar` files to the Hadoop cluster (YARN) and monitors the catastrophic execution, handling failures automatically."]},{"type":"heading","content":"The Main Components (Data Models)"},{"type":"paragraph","content":"Pig utilizes a highly flexible, nested mathematical data model, completely unlike rigid SQL tables:"},{"type":"list","items":["**Atom:** A single primitive value (e.g., an Integer `5`).","**Tuple:** An ordered set of fields, exactly like a row in SQL (e.g., `(5, \"John\")`).","**Bag:** An unordered collection of tuples. Crucially, a Bag can be placed *inside* another Tuple, mathematically allowing infinite nested hierarchies of data, which is completely impossible in standard SQL."]}]}},{"id":"v2-big-data-analytics-2026-pC-q5","number":22,"marks":10,"tags":["Part C"],"text":[{"type":"paragraph","content":"Explain Collaborative filtering in Recommender Systems."}],"answer":{"summary":"A massive mathematical analysis of Collaborative Filtering in Recommender Systems. Violently dissects User-Based vs. Item-Based approaches, detailing the catastrophic cold start problem and the elite execution of Matrix Factorization via Singular Value Decomposition (SVD).","explanation":[{"type":"heading","content":"Collaborative Filtering in Recommender Systems"},{"type":"paragraph","content":"Collaborative Filtering (CF) is the absolute mathematical backbone of massive enterprise recommendation engines (Netflix, Amazon). It completely ignores the physical attributes of the item (e.g., genre, director). Instead, it relies purely on the mathematical analysis of human behavioral data—the massive matrix of User-Item interactions (ratings, clicks, purchases)."},{"type":"heading","content":"1. Memory-Based Collaborative Filtering (The Neighborhood)"},{"type":"paragraph","content":"This approach mathematically calculates the similarity between users or items based strictly on historical ratings."},{"type":"subheading","content":"A. User-Based Collaborative Filtering"},{"type":"list","items":["**The Math:** If User Alice and User Bob have both rated 50 identical movies with the exact same 5-star scores, the algorithm uses Cosine Similarity or Pearson Correlation to mathematically define them as \"Highly Similar Neighbors\".","**The Execution:** If Bob watches a new movie and rates it 5 stars, the algorithm violently forces that movie into Alice's feed.","**The Catastrophic Demerit:** Users are chaotic and change their minds daily. Furthermore, computing the correlation between 100 Million Users against 100 Million Users requires an $N \times N$ matrix calculation, which mathematically crashes standard servers."]},{"type":"subheading","content":"B. Item-Based Collaborative Filtering (The Amazon Model)"},{"type":"list","items":["**The Math:** Invented by Amazon to solve the $N \times N$ scaling crisis. Instead of comparing chaotic users, it mathematically compares stable Items. If 90% of the people who bought \"Camera A\" also bought \"Battery B\", the algorithm creates a rigid mathematical link between the two physical items.","**The Merit:** Items are static. The similarity matrix can be computed entirely offline in Hadoop batch processing, and served in 5 milliseconds on the website."]},{"type":"heading","content":"2. Model-Based Collaborative Filtering (Matrix Factorization)"},{"type":"paragraph","content":"The absolute elite mathematical architecture, made famous by the $1 Million Netflix Prize."},{"type":"list","items":["**The Problem:** The User-Item matrix is massively \"Sparse\". Netflix has 10,000 movies, but a user only watches 50. 99.5% of the matrix is mathematically blank (missing data).","**Singular Value Decomposition (SVD):** A violent linear algebra algorithm. It mathematically shatters the massive, sparse matrix into two smaller, dense matrices: A User-Latent-Feature matrix and an Item-Latent-Feature matrix.","**Latent Features:** The algorithm mathematically discovers hidden dimensions (e.g., it might figure out a mathematical dimension for \"Dark Comedy\", even though no human ever labeled it).","**The Prediction:** To predict what Alice will rate Movie X, the algorithm simply executes a mathematical Dot Product of Alice's vector and Movie X's vector. This generates incredibly precise predictions even for sparse datasets."]},{"type":"heading","content":"The Catastrophic Flaw: The Cold Start Problem"},{"type":"paragraph","content":"Collaborative Filtering mathematically requires historical data to function. If a brand new movie is uploaded today, it has zero ratings. It physically cannot be linked to any item or user. The CF algorithm is mathematically blind to it. To solve this, architectures must use a Hybrid approach, blending CF with Content-Based filtering to recommend the new movie based on its metadata until enough human ratings are collected."}]}}]}],"loading":false,"error":null}}}

Q2Big Data Analytics

Question

Answer