Q20Machine Learning

Despite its name, Logistic Regression is an absolute foundational Binary Classification algorithm. A standard Linear Regression outputs a line $z={\theta }^{T}x$. For classification, this output must be violently squashed strictly into a probability range $[0,1]$. This is achieved by passing $z$ through the Sigmoid (Logistic) activation function:

$$\hat{y}=h_{\theta }(x)=\frac{1}{1+e^{-{\theta }^{T}x}}$$

If an engineer foolishly attempts to use the standard Linear Regression Cost Function (MSE) for Logistic Regression, the math fails catastrophically. Because the Sigmoid function is highly non-linear, inserting it into the MSE equation creates a horrific, non-convex cost function surface. It will contain thousands of "Local Minima". Gradient Descent will instantly get trapped, mathematically failing to ever find the true global minimum.

To solve this, the architecture utilizes Maximum Likelihood Estimation (MLE).

The probability of the target $y$ (where $y\in \{0,1\}$) given the input $x$ can be mathematically written as a single Bernoulli distribution equation:

$$P(y|x;\theta )=(\hat{y}{)}^y\cdot (1-\hat{y}{)}^{(1-y)}$$

• If the true label $y=1$, the right half mathematically vanishes ($somethin{g}^0=1$), leaving only the probability $P=\hat{y}$.
• If the true label $y=0$, the left half vanishes, leaving $P=(1-\hat{y})$.

To find the optimal weights $\theta$, we must maximize the total likelihood of the entire dataset. Because multiplying thousands of probabilities causes a catastrophic floating-point underflow in the CPU, we apply a mathematical logarithm to convert the multiplication into addition (Log-Likelihood). We then multiply by $-1$ to convert the maximization problem into a standard Minimization problem (Gradient Descent requires a loss to minimize).

This strictly derives the absolute Binary Cross-Entropy (Log-Loss) Cost Function for a single data point:

$$J(\theta )=-[y\log (\hat{y})+(1-y)\log (1-\hat{y})]$$

This newly derived function is mathematically guaranteed to be perfectly Convex (a single smooth bowl shape), completely eradicating the Local Minima crisis.

To minimize the cost, Gradient Descent mathematically calculates the partial derivative (gradient) of the massive cost function with respect to every single weight ${\theta }_j$. Through the brutal application of the Chain Rule of calculus across the Log and Sigmoid functions, the derivative beautifully collapses into an incredibly simple mathematical equation:

$$\frac{\partial J(\theta )}{\partial {\theta }_j}=\frac{1}{m}\sum _{i=1}^m({\hat{y}}^{(i)}-y^{(i)})x_j^{(i)}$$

The algorithm then violently updates the weights using the learning rate $\alpha$:

$${\theta }_j:={\theta }_j-\alpha \frac{\partial J(\theta )}{\partial {\theta }_j}$$

This loop executes relentlessly until the gradient mathematically reaches zero (Convergence at the absolute global minimum).