Courses
MDE-MET-01 Calculus and Linear Algebra for Graduate Students
Matrices

Matrices

A matrix is a rectangular array of numbers.

Matrices are usually denoted by uppercase letters.

A matrix, $A$, with $m$ rows and ${n}$ columns, is refered to as an ${m\times n}$ matrix.

If all the entries are real numbers, the matrix is a member of the set of all real matrices with $m$ rows and ${n}$ columns, i.e. $A \in\mathbb{R}^{m \times n}$.

The entry in the $i$th row and $j$th column of the matrix $A \in \mathrm{R}^{m \times n}$ is denoted by $a_{ij}$ or $a_{i,j}$. Hence, the matrix is written as

$$ \begin{align*} A = \left( \begin{array}{cccc} a_{1,1} & a_{1,2} & \cdots & a_{1,n} \\ a_{2,1} & a_{2,2} & \cdots & a_{2,n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m,1} & a_{m,2} & \cdots & a_{m,n} \end{array} \right). \end{align*} $$

A matrix is said to be a square matrix if the number of rows is equal to the number of columns, i.e. $n=m$.

Matrix Operations

Definition: Matrix Matrix Multiplication

For two matrices $A \in \mathbb{R}^{m \times n}$ and $B \in \mathbb{R}^{n \times l}$, the matrix product $C=AB$ is a $m \times l$ matrix whose entries are given by

$$ c_{ij} = \sum_{k=1}^{n} a_{ik} b_{kj} $$

for any $i=1, \ldots, m$ and $j =1, \ldots, l$.

If $A$ is an $m\times n$ matrix and $B$ is a $n\times l$ matrix, then $A$ has the same number of columns as $B$ has rows, i.e. $n$. Thus, each entry $c_{ij}$ is the dot product of the $i$th row of $A$ with the $j$th column of $B$, which both have length $n$.

Definition: Matrix Vector Multiplication

For a matrix $A \in \mathbb{R}^{m \times n}$ and vector $\boldsymbol{x} \in \mathbb{R}^{n}$, the matrix-vector product $A\boldsymbol{x} =\boldsymbol{y}$ is an $m \times l$ vector whose entries are given by

$$ y_{j} = \sum_{i=1}^{n} a_{ij} x_i . $$
Definition: Row and Column Vectors
When matrix $A \in \mathbb{R}^{m \times 1}$ is comprised of a single column of $m$ entries, it is called a column vector. Similarly, a $1 \times n$ matrix, i.e. a single row of $n$ entries is called a row vector.

Thus $\boldsymbol{v}^T \boldsymbol{v}$ is a $1 \times 1$ matrix, and $\boldsymbol{v} \boldsymbol{v}^T$ is a $n \times n$ matrix.

Definition: Transpose
For a matrix $A \in \mathbb{R}^{m \times n}$, the transpose of the matrix $A^T \in \mathbb{R}^{n \times m}$ where all elements are mapped to $a_{ij} = a_{ji}$.

Thus, if $\boldsymbol{x}$ is a column vector it is possible to perform $A\boldsymbol{x}=\boldsymbol{y}$, then the equivalent multiplication by a row vector is $\boldsymbol{x}^T A^T = \boldsymbol{y}^T$.

Definition: Trace

For a square matrix $A \in \mathbb{R}^{n \times n}$, the trace is defined as the sum of the elements on the main diagonal.

$$ \text{tr}\left(A \right) = \sum_{i=0}^{n} a_{ii}. $$

Properties of Matrix Operations

  1. $A+B = B+A$
  2. $c\left(A + B\right) = cA + cB$
  3. $C\left(A + B\right) = CA + CB$
  4. $\left(A + B\right)C = AC + BC$
  5. $AB \neq BA$
  6. $A + \left(B + C\right) = \left(A+ B\right) + C$
  7. $A\left( BC \right) = \left(AB\right)C$
Proposition:
$\left( A B \right)^T = B^T A^T$

Useful Types of Matrices

  • The identity matrix is the square matrix $I$ such that $AI=A$ and $IA=A$, all entries on the main diagonal are one, all others are zero.

  • A diagonal matrix is a matrix where all entries outside the main diagonal are all zero, i.e. $a_{ij} =0$ when $i \neq j$.

  • A banded matrix is a matrix whose non-zero entries are confined to a diagonal band, comprising the main diagonal and zero or more diagonals on either side.

  • The inverse matrix of a square matrix is the matrix $A^{-1}$ such that $AA^{-1}~=~A^{-1}A~=~I.$ A matrix is non-singular or invertible if there exists an inverse matrix exists.

A square matrix has an inverse if and only if the rank of the matrix is equal to the number of columns (or rows), equivalently, the columns are linearly independent. Such matrices are said to have full rank.

  • A square matrix $A$ is symmetric if ${A=A^T}$, that is, $a_{i,j}=a_{j,i}$ for all indices $i$ and $j$.

  • A square matrix, with complex elements, is said to be Hermitian if the matrix is equal to its conjugate transpose, i.e. $a_{i,j}=\overline{a_{j,i}}$ for all indices $i$ and $j$. A Hermitian matrix is written as $A=A^H$.

  • An orthogonal matrix $Q$ is a matrix whose columns $\vec{q}_i$ are orthogonal to one another, that is $\vec{q}_i \cdot \vec{q}_j = 0$ for $i \neq j$ and have unit length, i.e. $\left\| \vec{q}_i \right\| = 1$.

Proposition:
The inverse of an orthogonal matrix $Q$ is equal to its transpose, i.e. $Q^{-1} = Q^T$.

  • A permutation matrix is a square matrix that has exactly one entry of $1$ in each row and each column and all other entries $0$.

  • Rotation matrices describe rotations by an angle about the axes of a coordinate system. They can be denoted by $R_x ( \alpha )$, i.e. rotation by angle $\theta$ about the $x$ axis. The product of two rotation matrices is also a rotation matrix. Note that in general $R_x\left( \theta \right) R_y \left( \beta \right) \neq R_x\left( \theta \right) R_y \left( \beta \right)$.

  • Reflection matrices: $R = I - 2 \boldsymbol{u} \boldsymbol{u}^T$.

  • A square matrix is said to be lower triangular matrix if all the elements above the main diagonal are zero, i.e. $a_{ij}=0$ when $i < j$. Similarly, a matrix is said to be upper triangular if all the entries below the main diagonal are zero, that is $a_{ij}=0$ when $i~>~j$. A matrix is said to be strictly upper triangular (or strictly lower triangle) if the main diagonal is also zero.

Properties of Nonsingular Matrices

For a nonsingular matrix, the following all hold:

  • A nonsingular matrices has full rank, i.e. the rank is equal to the number of rows or columns

  • A square matrix is nonsingular if and only if the determinant of the matrix is non-zero.

  • If a matrix is singular, both versions of Gaussian elimination (with and without pivoting) will fail due to division by zero, yielding a floating exception error. Another way to understand this is that the number of pivots is equal to the rank, so if the matrix does not have full rank, so there will not be enough pivots in order to transform the matrix in to row-echelon form.