Q22Big Data Analytics

Collaborative Filtering (CF) is the absolute mathematical backbone of massive enterprise recommendation engines (Netflix, Amazon). It completely ignores the physical attributes of the item (e.g., genre, director). Instead, it relies purely on the mathematical analysis of human behavioral data—the massive matrix of User-Item interactions (ratings, clicks, purchases).

This approach mathematically calculates the similarity between users or items based strictly on historical ratings.

A. User-Based Collaborative Filtering

• The Math: If User Alice and User Bob have both rated 50 identical movies with the exact same 5-star scores, the algorithm uses Cosine Similarity or Pearson Correlation to mathematically define them as "Highly Similar Neighbors".
• The Execution: If Bob watches a new movie and rates it 5 stars, the algorithm violently forces that movie into Alice's feed.
• The Catastrophic Demerit: Users are chaotic and change their minds daily. Furthermore, computing the correlation between 100 Million Users against 100 Million Users requires an $NimesN$ matrix calculation, which mathematically crashes standard servers.

B. Item-Based Collaborative Filtering (The Amazon Model)

• The Math: Invented by Amazon to solve the $NimesN$ scaling crisis. Instead of comparing chaotic users, it mathematically compares stable Items. If 90% of the people who bought "Camera A" also bought "Battery B", the algorithm creates a rigid mathematical link between the two physical items.
• The Merit: Items are static. The similarity matrix can be computed entirely offline in Hadoop batch processing, and served in 5 milliseconds on the website.

The absolute elite mathematical architecture, made famous by the $1 Million Netflix Prize.

• The Problem: The User-Item matrix is massively "Sparse". Netflix has 10,000 movies, but a user only watches 50. 99.5% of the matrix is mathematically blank (missing data).
• Singular Value Decomposition (SVD): A violent linear algebra algorithm. It mathematically shatters the massive, sparse matrix into two smaller, dense matrices: A User-Latent-Feature matrix and an Item-Latent-Feature matrix.
• Latent Features: The algorithm mathematically discovers hidden dimensions (e.g., it might figure out a mathematical dimension for "Dark Comedy", even though no human ever labeled it).
• The Prediction: To predict what Alice will rate Movie X, the algorithm simply executes a mathematical Dot Product of Alice's vector and Movie X's vector. This generates incredibly precise predictions even for sparse datasets.

Collaborative Filtering mathematically requires historical data to function. If a brand new movie is uploaded today, it has zero ratings. It physically cannot be linked to any item or user. The CF algorithm is mathematically blind to it. To solve this, architectures must use a Hybrid approach, blending CF with Content-Based filtering to recommend the new movie based on its metadata until enough human ratings are collected.