Covariance Matrix

A covariance matrix is a type of matrix used to describe the covariance values between two items in a random vector. It is also known as the variance-covariance matrix because the variance of each element is represented along the matrix’s major diagonal, and the covariance is represented among the non-diagonal elements.

• It is particularly important in fields like data science, machine learning, and finance, where understanding relationships between multiple variables is crucial.
• Covariance matrices play a key role in stochastic modeling and principal component analysis (PCA).

They help us identify the direction of relationships (positive or negative) between variables and are essential for understanding and analyzing high-dimensional datasets.

Mathematical Definition

The variance-covariance matrix is a square matrix with diagonal elements that represent the variance and the non-diagonal components that express covariance.

• The covariance of a variable can take any real value- positive, negative, or zero.
• A positive covariance suggests that the two variables have a positive relationship, whereas a negative covariance indicates that they do not.
• If two elements do not vary together, they have a zero covariance.

Example: Let's say there are 2 data sets, X = [10, 5] and Y = [3, 9]. The variance of set X = 6.25, and the variance of set Y = 9. The covariance between both variables is −15. The covariance matrix is as follows:

\begin{bmatrix} Variance~of~Set~X & Covariance~of~Both~Sets\\ Covariance~of~Both~Sets& Variance~of~Set~Y \end{bmatrix}=\begin{bmatrix} 6.25 & -15\\ -15& 9 \end{bmatrix}

Formula

The general form of a covariance matrix is given as follows:

\begin{bmatrix} \mathrm{Var}(x_1) & \cdots & \mathrm{Cov}(x_n, x_1) \\ \vdots & \ddots & \vdots \\ \mathrm{Cov}(x_n, x_1) & \cdots & \mathrm{Var}(x_n) \end{bmatrix}

Where,

• Sample Variance: var(x₁) = \frac{\sum_{1}^{n}\left ( x_{i} -\overline{x}\right )^{2} }{n-1}
• Sample Covariance: cov(x₁, y₁) = \frac{\sum_{1}^{n}\left (x_{i} -\overline{x}\right )\left(y_{i}-\overline{y}\right)}{n-1}
• Population Variance: var(x_n) = \frac{\sum_{1}^{n}\left ( x_{i} -\mu\right )^{2} }{n}
• Population Covariance: cov(x_n, y_n) = \frac{\sum_{1}^{n}\left ( x_{i} -\mu_{x}\right )\left ( y_{i}-\mu_{y} \right ) }{n}

Here,

• μ is the Mean of the Population
• \overline x is the Mean of the Sample
• n is the Number of observations
• x_i is the Observation in Dataset x

2 ⨯ 2 Covariance Matrix

We know that in a 2 ⨯ 2 matrix there are two rows and two columns.

Hence, the 2 ⨯ 2 Covariance Matrix can be expressed as \begin{bmatrix}\mathrm{var(x)}& \mathrm{cov(x,y)} \\\mathrm{cov(x,y)} &\mathrm{var(y)}\end{bmatrix}

3 ⨯ 3 Covariance Matrix

In a 3⨯3 Matrix, there are 3 rows and 3 columns. We know that in a Covariance Matrix, the diagonal elements are variance, and the non-diagonal elements are covariance.

Hence, a 3⨯3 Covariance Matrix can be given as \begin{bmatrix}\mathrm{var(x)}&\mathrm{cov(x,y)} &\mathrm{cov(x,z)} \\\mathrm{cov(x,y)} &\mathrm{var(y)} &\mathrm{cov(y,z)} \\\mathrm{cov(x,z)} &\mathrm{cov(y,z)} &\mathrm{var(z)} \\\end{bmatrix}

Methods to Calculate Covariance Matrix

The dimensions of a covariance matrix are determined by the number of variables in a given data set. If there are only two variables in a set, then the covariance matrix would have two rows and two columns. Similarly, if a data set has three variables, then its covariance matrix would have three rows and three columns. 

The data pertains to the marks scored by Anna, Caroline, and Laura in Psychology and History. Make a covariance matrix.

The following steps have to be followed:

Step 1: Find the mean of variable X. Sum up all the observations in variable X and divide the sum obtained with the number of terms. Thus, (80 + 63 + 100)/3 = 81.

Step 2: Subtract the mean from all observations. (80 - 81), (63 - 81), (100 - 81).

Step 3: Take the squares of the differences obtained above and then add them up. Thus, (80 - 81)² + (63 - 81)² + (100 - 81)².

Step 4: Find the variance of X by dividing the value obtained in Step 3 by 1 less than the total number of observations. var(X) = [(80 - 81)² + (63 - 81)² + (100 - 81)²] / (3 - 1) = 343.

Step 5: Similarly, repeat steps 1 to 4 to calculate the variance of Y. Var(Y) = 633.333

Step 6: Choose a pair of variables.

Step 7: Subtract the mean of the first variable (X) from all observations; (80 - 81), (63 - 81), (100 - 81).

Step 8: Repeat the same for variable Y; (70 - 47), (20 - 47), (50 - 47).

Step 9: Multiply the corresponding terms: (80 - 81)(70 - 47), (63 - 81)(20 - 47), (100 - 81)(50 - 47).

Step 10: Find the covariance by adding these values and dividing them by (n - 1). Cov(X, Y) = [(80 - 81)(70 - 47) + (63 - 81)(20 - 47) + (100 - 81)(50 - 47)]/(3-1) = 260.

Step 11: Use the general formula for the covariance matrix to arrange the terms. The matrix becomes: \begin{bmatrix} 343 & 260\\ 260& 633.333 \end{bmatrix}

Real-Life Applications

• Stock Market - Investors use a covariance matrix to see how different stocks move together.
• Weather Forecasting - Meteorologists use it to study how temperature, humidity, and wind speed are related over time.
• Image Processing - In facial recognition, covariance helps us to identify patterns and differences between images by analyzing pixel relations.
• Economics - Economists use to analyse how different economic indicators move together.

Practice Questions on Covariance Matrix