Applications of Linear Algebra


We will provide a practical review of linear algebra concepts and usage while skipping some introduction to linear algebra operations.

The following topics are discussed in this article: inner product, norm, eigenvalue decomposition, diagonalization of matrices, matrix norm

Inner Product

Definition: An inner product on a vector space is a function that takes two vectors and returns a scalar, typically denoted asx,y. It generalizes the dot product in Euclidean space.

Examples and Applications: 


Probability and Statistics:

The expectation of a random variable X, denoted E[X], can be interpreted as an inner product between the range of X and its associated probability density function PDF. The result of this expectation is a scalar value which in this case is the average value that one expects this random variable X will take.

Signal Processing - Convolution Operation:


Definition: The norm of a vector x, denoted asx, is a measure of the 'length' of the vector. In an inner product space, it is defined as

x∥= ⟨x, x

Eigenvalue Decomposition

Concept: If A is a square matrix, an eigenvalue decomposition finds matrices P and D such that A=PDP ^−1 where D is diagonal and P contains the eigenvectors of A. Eigenvalues and Eigenvectors: An eigenvalue λ and eigenvector v of A satisfy Av=λv.

Applications: Used in solving linear differential equations, PCA in machine learning, and more.

Diagonalization of Matrices

Definition: A matrix A is diagonalizable if it can be transformed into a diagonal matrix through a similarity transformation (i.e., preservation of eigenvalues before and after the transformation). This means there exists a matrix P such that performing the operation P^−1AP results in a diagonal matrix, which we will call E. In E, only the diagonal elements contain values, and all other elements are zeros.

Note: In linear algebra, a transformation typically refers to changing the coordinate system through the multiplication by a transformation matrix. This process alters the representation of vectors or matrices without changing their intrinsic properties. Transformation Matrix: The matrix P is used to perform this transformation. It acts as a tool to shift from one coordinate system to another.

A specific type of transformation is similarity transformation, which is a process where a matrix  A is converted into another matrix B using a particular form of matrix multiplication, involving a matrix P and its inverse. This process is described by the formula B=P^−1AP.

Condition: A is diagonalizable if it has n linearly independent eigenvectors, where n is the dimension of the matrix 

Implication and Applications


Matrix Norm

Definition: A matrix norm is a natural extension of vector norms to matrices, providing a measure of the 'size' of the matrix.