Q11Machine Learning

In Machine Learning, if a Linear Regression model is fed a dataset with 500 features and only 100 rows, it will mathematically memorize the noise (Overfitting). The model's weight vector $\theta$ will explode to massive values (e.g., $w_1=45000$, $w_2=-44900$) just to perfectly fit the training points. Regularization is the absolute architectural solution to this catastrophe; it mathematically penalizes massive weights by modifying the Cost Function $J(\theta )$.

1. Ridge Regularization (L2 Penalty)

Ridge aggressively adds the squared magnitude of the weights to the loss function.

• Mathematical Equation: $J(\theta )=\text{MSE}+\lambda {\sum }_{i=1}^n{\theta }_i^2$
• Mechanism: The hyperparameter $\lambda$ dictates the violence of the penalty. Because the penalty is squared, a weight of $10$ incurs a massive penalty of $100$. The gradient descent algorithm is mathematically forced to shrink all weights smoothly toward zero to minimize this penalty.
• Result: All features remain in the model, but their impact is suppressed. It is exceptionally good at handling highly correlated features (multicollinearity).

2. Lasso Regularization (L1 Penalty)

Least Absolute Shrinkage and Selection Operator (Lasso) adds the absolute magnitude of the weights.

• Mathematical Equation: $J(\theta )=\text{MSE}+\lambda {\sum }_{i=1}^n|{\theta }_i|$
• Mechanism: The geometry of the L1 absolute value penalty creates a sharp mathematical "diamond" constraint space. When the loss function contour intersects this diamond, it almost always hits exactly on a corner.
• Result: Absolute Feature Selection. Lasso violently forces the weights of irrelevant features to be EXACTLY $0.0$. If you input 500 features, Lasso might output a model using only 12 features, creating a highly sparse, interpretable mathematical architecture.