Q1Advanced Engineering Mathematics

Question

If $θ$ be the acute angle between the two lines of regression of variables $x$ and $y$ , show that:

tan θ = (\frac{1 - r ^{2}}{r}) (\frac{σ _{x} σ _{y}}{σ _{x}^{2} + σ _{y}^{2}})

where $r$ , $σ_{x}$ , $σ_{y}$ have their usual meaning. Explain the significance where $r = 0$ and $r = \pm 1$ .

Answer 1

In statistical modeling, regression analysis is a powerful predictive technique that investigates the relationship between a dependent (target) and independent variable(s) (predictor). When studying two continuous random variables, $X$ and $Y$ , we often seek to understand not just if they are correlated, but how one can be used to mathematically predict the other. This gives rise to two distinct lines of regression: the regression of $Y$ on $X$ , and the regression of $X$ on $Y$ . The geometric angle between these two lines provides profound insights into the strength and nature of the linear correlation between the variables.

1. Fundamentals of Linear Regression

Let us consider a bivariate data set consisting of $n$ pairs of observations $(x_{1}, y_{1}), (x_{2}, y_{2}), \dots, (x_{n}, y_{n})$ . The primary objective of linear regression is to find the "best-fitting" straight line through these points. The universally accepted criterion for this "best fit" is the Principle of Least Squares, which minimizes the sum of the squared deviations (errors) of the observed data points from the line.

Because we can measure these deviations either vertically (errors in predicting $Y$ ) or horizontally (errors in predicting $X$ ), we obtain two distinct regression lines.

Regression Line of Y on X: This line is constructed by minimizing the vertical distances (errors in $Y$ ). It is mathematically used to predict the value of $Y$ for a given, fixed value of $X$ .
Regression Line of X on Y: Conversely, this line is constructed by minimizing the horizontal distances (errors in $X$ ). It is used to predict the value of $X$ for a given, fixed value of $Y$ .

2. Equations of the Regression Lines

Let $\overset{x}{ˉ}$ and $\overset{y}{ˉ}$ denote the arithmetic means of the $X$ and $Y$ series, respectively. Let $σ_{x}$ and $σ_{y}$ denote their respective standard deviations, and let $r$ represent the Pearson correlation coefficient between $X$ and $Y$ .

The fundamental equation for the regression line of $Y$ on $X$ is given by:

y - \overset{y}{ˉ} = b_{y x} (x - \overset{x}{ˉ})

Here, $b_{y x}$ is the regression coefficient of $Y$ on $X$ , which represents the slope of this line relative to the x-axis. It is calculated as:

b_{y x} = r \frac{σ _{y}}{σ _{x}}

Substituting $b_{y x}$ back into the equation, we get:

y - \overset{y}{ˉ} = r \frac{σ _{y}}{σ _{x}} (x - \overset{x}{ˉ}) — (Equation 1)

Similarly, the fundamental equation for the regression line of $X$ on $Y$ is given by:

x - \overset{x}{ˉ} = b_{x y} (y - \overset{y}{ˉ})

Where $b_{x y}$ is the regression coefficient of $X$ on $Y$ . It is calculated as:

b_{x y} = r \frac{σ _{x}}{σ _{y}}

Substituting $b_{x y}$ , we get:

x - \overset{x}{ˉ} = r \frac{σ _{x}}{σ _{y}} (y - \overset{y}{ˉ})

To analyze the geometric relationship between these lines in the standard Cartesian coordinate system (where $y$ is a function of $x$ ), we must rearrange this second equation into the standard slope-intercept form ( $y = m x + c$ ). Solving for $y$ , we get:

y - \overset{y}{ˉ} = \frac{1}{r} \frac{σ _{y}}{σ _{x}} (x - \overset{x}{ˉ}) — (Equation 2)

3. Deriving the Angle Between the Lines ( $θ$ )

From Equation 1 and Equation 2, we can identify the true mathematical slopes of the two regression lines:

Slope of the line of $Y$ on $X$ : $m_{1} = r \frac{σ _{y}}{σ _{x}}$
Slope of the line of $X$ on $Y$ : $m_{2} = \frac{1}{r} \frac{σ _{y}}{σ _{x}}$

From basic coordinate geometry, if $θ$ is the acute angle between two intersecting straight lines with slopes $m_{1}$ and $m_{2}$ , the tangent of this angle is given by the well-known formula:

tan θ = \frac{m _{2} - m _{1}}{1 + m _{1} m _{2}}

We now substitute our expressions for $m_{1}$ and $m_{2}$ into this trigonometric identity:

tan θ = \frac{\frac{1}{r} \frac{σ _{y}}{σ _{x}} - r \frac{σ _{y}}{σ _{x}}}{1 + ( r \frac{σ _{y}}{σ _{x}} ) ( \frac{1}{r} \frac{σ _{y}}{σ _{x}} )}

Let us carefully simplify the numerator by factoring out the common term $\frac{σ _{y}}{σ _{x}}$ :

Numerator = \frac{σ _{y}}{σ _{x}} (\frac{1}{r} - r) = \frac{σ _{y}}{σ _{x}} (\frac{1 - r ^{2}}{r})

Now, let us simplify the denominator. Notice that the correlation coefficient $r$ perfectly cancels out in the multiplication:

Denominator = 1 + (r \frac{σ _{y}}{σ _{x}}) (\frac{1}{r} \frac{σ _{y}}{σ _{x}}) = 1 + \frac{σ _{y}^{2}}{σ _{x}^{2}} = \frac{σ _{x}^{2} + σ _{y}^{2}}{σ _{x}^{2}}

Putting the simplified numerator and denominator back into the tangent formula:

tan θ = \frac{\frac{σ _{y}}{σ _{x}} ( \frac{1 - r ^{2}}{r} )}{\frac{σ _{x}^{2} + σ _{y}^{2}}{σ _{x}^{2}}}

tan θ = \frac{σ _{y}}{σ _{x}} (\frac{1 - r ^{2}}{r}) \times \frac{σ _{x}^{2}}{σ _{x}^{2} + σ _{y}^{2}}

tan θ = \frac{1 - r ^{2}}{r} \times \frac{σ _{x} σ _{y}}{σ _{x}^{2} + σ _{y}^{2}}

Since standard deviations $σ_{x}$ and $σ_{y}$ are always strictly positive, and we are looking for the acute angle (meaning $tan θ$ must be positive), we can write the final canonical formula as:

θ = tan^{- 1} [\frac{1 - r ^{2}}{∣ r ∣} (\frac{σ _{x} σ _{y}}{σ _{x}^{2} + σ _{y}^{2}})]

4. Critical Analysis and Special Cases

This derived formula is remarkably elegant because it directly links the geometric separation of the lines to the statistical strength of their correlation. Let us deeply analyze the extreme cases to understand its physical significance.

Case A: Perfect Linear Correlation ( $r = + 1$ or $r = - 1$ )

When the variables exhibit a perfect linear relationship, all data points lie exactly on a single straight line. In this scenario, $r^{2} = 1$ . Substituting this into our formula yields $1 - r^{2} = 0$ . Therefore, $tan θ = 0$ , which implies that the angle $θ = 0$ . Geometrically, this means the angle between the two regression lines vanishes; they completely coincide and become identical. There is no distinction between minimizing vertical errors versus horizontal errors because there are zero errors.

Case B: Absolutely No Linear Correlation ( $r = 0$ )

When the variables are completely uncorrelated linearly, $r = 0$ . Substituting this into our formula causes the denominator of the first fraction to become zero ( $∣ r ∣ \to 0$ ). As a result, the value of $tan θ$ approaches infinity ( $tan θ \to \infty$ ). The angle whose tangent is infinity is $\frac{π}{2}$ radians, or 90 degrees. Geometrically, the two regression lines are perfectly perpendicular (orthogonal) to each other. Specifically, they align parallel to the X and Y coordinate axes, indicating that knowledge of one variable provides absolutely no predictive power over the other.

5. Graphical Representation

The relationship between the correlation coefficient $r$ and the angle $θ$ between the regression lines can be visualized in the following diagram.

Q1Advanced Engineering Mathematics

Question

Answer

1. Fundamentals of Linear Regression

2. Equations of the Regression Lines

3. Deriving the Angle Between the Lines (θ)

4. Critical Analysis and Special Cases

5. Graphical Representation

3. Deriving the Angle Between the Lines ( $θ$ )