mean x SQUARED

population variance

$$(a+b) 2 =a 2 +b 2 +2ab$$

Var(Z 1) )=α 1⊤ Σα 1 =α 1⊤(λα 1)=λ⋅α 1⊤ α 1 =λ⋅1=λ

Derivative of alpha1Xalpha1' = 2 x alpha1

• length needs to be 1 = unit vector
• so that variance cannot be arbitrary large
• we want the TRUE largest variance
• the notation here is confusing but it is the same as alpha1 x alpha1

Proof that Var(Z1) = alpha^T * Sigma * alpha (3D case)

We want to show that the variance of a linear combination of random variables can be written in matrix notation:

$$ \text{Var}(Z_1) = \boldsymbol{\alpha}^T \Sigma \boldsymbol{\alpha} $$

for a vector $\boldsymbol{\alpha} = (\alpha_1, \alpha_2, \alpha_3)^T$, and $\Sigma$ the covariance matrix of the variables $X_1, X_2, X_3$.

Step-by-Step Expansion

Let:

• $Z_1 = \alpha_1 X_1 + \alpha_2 X_2 + \alpha_3 X_3$
• $\boldsymbol{\alpha} = (\alpha_1, \alpha_2, \alpha_3)^T$
• $\Sigma$ is the 3x3 covariance matrix:

$$ \Sigma = \begin{bmatrix} \sigma_{11} & \sigma_{12} & \sigma_{13} \ \sigma_{21} & \sigma_{22} & \sigma_{23} \ \sigma_{31} & \sigma_{32} & \sigma_{33} \end{bmatrix} $$

Since $\Sigma$ is a covariance matrix, it is symmetric:

• $\sigma_{ij} = \text{Cov}(X_i, X_j)$
• $\sigma_{ii} = \text{Var}(X_i)$

Step 1: Expand Var(Z1)

$$ \text{Var}(Z_1) = \text{Var}(\alpha_1 X_1 + \alpha_2 X_2 + \alpha_3 X_3) $$

Using the variance formula for linear combinations:

$$ = \alpha_1^2 \text{Var}(X_1) + \alpha_2^2 \text{Var}(X_2) + \alpha_3^2 \text{Var}(X_3) + 2\alpha_1\alpha_2 \text{Cov}(X_1, X_2) + 2\alpha_1\alpha_3 \text{Cov}(X_1, X_3) + 2\alpha_2\alpha_3 \text{Cov}(X_2, X_3) $$

Step 2: Compute $\boldsymbol{\alpha}^T \Sigma \boldsymbol{\alpha}$

Write out the multiplication:

$$ \boldsymbol{\alpha}^T \Sigma \boldsymbol{\alpha} = \begin{bmatrix} \alpha_1 & \alpha_2 & \alpha_3 \end{bmatrix} \begin{bmatrix} \sigma_{11} & \sigma_{12} & \sigma_{13} \ \sigma_{21} & \sigma_{22} & \sigma_{23} \ \sigma_{31} & \sigma_{32} & \sigma_{33} \end{bmatrix} \begin{bmatrix} \alpha_1 \ \alpha_2 \ \alpha_3 \end{bmatrix} $$

First compute $\Sigma \boldsymbol{\alpha}$:

$$ = \begin{bmatrix} \sigma_{11}\alpha_1 + \sigma_{12}\alpha_2 + \sigma_{13}\alpha_3 \ \sigma_{21}\alpha_1 + \sigma_{22}\alpha_2 + \sigma_{23}\alpha_3 \ \sigma_{31}\alpha_1 + \sigma_{32}\alpha_2 + \sigma_{33}\alpha_3 \end{bmatrix} $$

Then multiply with $\boldsymbol{\alpha}^T$:

$$ \boldsymbol{\alpha}^T (\Sigma \boldsymbol{\alpha}) = \alpha_1(\sigma_{11}\alpha_1 + \sigma_{12}\alpha_2 + \sigma_{13}\alpha_3) + \alpha_2(\sigma_{21}\alpha_1 + \sigma_{22}\alpha_2 + \sigma_{23}\alpha_3) + \alpha_3(\sigma_{31}\alpha_1 + \sigma_{32}\alpha_2 + \sigma_{33}\alpha_3) $$

Expanding:

$$ = \alpha_1^2 \sigma_{11} + \alpha_1\alpha_2 \sigma_{12} + \alpha_1\alpha_3 \sigma_{13} + \alpha_2\alpha_1 \sigma_{21} + \alpha_2^2 \sigma_{22} + \alpha_2\alpha_3 \sigma_{23} + \alpha_3\alpha_1 \sigma_{31} + \alpha_3\alpha_2 \sigma_{32} + \alpha_3^2 \sigma_{33} $$

Group terms:

• Diagonal terms (variances): $\alpha_1^2 \sigma_{11} + \alpha_2^2 \sigma_{22} + \alpha_3^2 \sigma_{33}$
• Off-diagonal terms (covariances): $2\alpha_1\alpha_2\sigma_{12} + 2\alpha_1\alpha_3\sigma_{13} + 2\alpha_2\alpha_3\sigma_{23}$

✅ Final Conclusion:

$$ \boldsymbol{\alpha}^T \Sigma \boldsymbol{\alpha} = \alpha_1^2 \text{Var}(X_1) + \alpha_2^2 \text{Var}(X_2) + \alpha_3^2 \text{Var}(X_3) + 2\alpha_1 \alpha_2 \text{Cov}(X_1, X_2) + 2\alpha_1 \alpha_3 \text{Cov}(X_1, X_3) + 2\alpha_2 \alpha_3 \text{Cov}(X_2, X_3) $$

Which proves the formula!

• Z1 usually captures the the direction with the most variance or spread
• The new coordinate system is rotated such that both variables dont contain info about each other anymore

PCA: Co-movements vs Counter-movements

Z₁ (Co-movements)

α₁ = [ 0.7048 ] [ 0.7094 ]

• Both values are positive
• When X₁ and X₂ both increase or both decrease, their effects add up
• This makes Z₁ large in magnitude

✅ Z₁ captures co-movements: when X₁ and X₂ move in the same direction

Z₂ (Counter-movements)

α₂ = [ 0.7094 ] [ -0.7048 ]

• One value is positive, the other is negative
• When X₁ increases and X₂ decreases (or vice versa), their effects combine
• This makes Z₂ large

✅ Z₂ captures counter-movements: when X₁ and X₂ move in opposite directions

🔹 set.seed(20) This makes the random number generation repeatable If you run the code again, you'll get the same random numbers Useful for reproducibility (e.g., in class, exams, or reports) 🔹 w <- rnorm(1000, 0, 1) This creates a variable w with 1000 random numbers From a normal distribution with: Mean = 0 Standard deviation = 1 These are your "base" values, which will be used to create x1 and influence x2 🔹 x <- data.frame(...) This creates a new data frame x with two columns:

✅ x1 = w

Just copies the w values directly So x1 is standard normal ✅ x2 = 0.8 * w + sqrt(1 - 0.8^2) * rnorm(1000)

This is the key part! It creates a new variable x2 that is:

Correlated with x1 Still normally distributed Let’s break it down:

0.8 * w: This makes x2 partly depend on w — 80% influence → introduces correlation sqrt(1 - 0.8^2): This is Square root (1−0.64=0.36)

It ensures that total variance of x2 stays 1 rnorm(1000): Generates another 1000 random numbers from a standard normal distribution This adds the random noise part, independent of w So the full formula is:

x2=0.8⋅x1+0.6⋅random noise This creates a variable x2 that:

Looks normal Has correlation 0.8 with x1 Has variance 1

z1 and z2 represent coordinates for point A in red system * We see that z1 then should have a negative sign even tho it is pointed to the right * because alpha 1 is pointed to the left

Using linear algebra is also possible like orthogonal projection * proj v (a) = (a x v / v x v) x (v) 1. Use the formula for two points 2. Find distance between origin and A'

1. We want to make sure X2 also has variance of 1
2. W1 is X1 and W2 is random noises which is independent from W1
3. put X2 in the whole variance formula then you see that -->
4. Var(X2) = p^2Var(W1) + homeports of Var(W2)
5. But Var(X2) = 1 then we know there is only 1-p^2 part left for variance W2
6. Put it back in the function required square root !