The Euclidean space¶

Inner product
Norm
Distance
\(\ell_p\) norms

In this book we will be generally concerned with the Euclidean space \(\RR^N\). This section summarizes important results for this space.

\(\RR^2\) (the 2-dimensional plane) and \(\RR^3\) the 3-dimensional space are the most familiar spaces to us.

\(\RR^N\) is a generalization in \(N\) dimensions.

Definition

Let \(\RR\) denote the field of real numbers. For any positive integer \(N\), the set of all \(N\)-tuples of real numbers forms an \(N\)-dimensional vector space over \(\RR\) which is denoted as \(\RR^N\) and sometimes called real cooordinate space.

An element \(x\) in \(\RR^N\) is written as

\[x = (x_1, x_2, \ldots, x_N),\]

where each \(x_i\) is a real number.

Vector space operations on \(\RR^N\) are defined by:

\[\begin{split}&x + y = (x_1 + y_1, x_2 + y_2, \dots, x_N + y_N), \quad \forall x, y \in \RR^N.\\ & \alpha x = (\alpha x_1, \alpha x_2, \dots, \alpha x_N) \quad \forall x \in \RR^N, \alpha \in \RR .\end{split}\]

\(\RR^N\) comes with the standard ordered basis \(B = \{e_1, e_2, \dots, e_N\}\):

(1)¶\[\begin{split}\begin{aligned} & e_1 = (1,0,\dots, 0),\\ & e_2 = (0,1,\dots, 0),\\ &\vdots\\ & e_N = (0,0,\dots, 1) \end{aligned}\end{split}\]

An arbitrary vector \(x\in\RR^N\) can be written as

\[x = \sum_{i=1}^{N}x_i e_i\]

Inner product ¶

Standard inner product (a.k.a. dot product) is defined as:

\[\langle x, y \rangle = \sum_{i=1}^{N} x_i y_i = x_1 y_1 + x_2 y_2 + \dots + x_N y_N \quad \forall x, y \in \RR^N.\]

This makes \(\RR^N\) an inner product space.

The result is always a real number. Hence we have symmetry:

\[\langle x, y \rangle = \langle y, x \rangle\]

Norm ¶

The length of the vector (a.k.a. Euclidean norm or \(\ell_2\) norm) is defined as:

\[\| x \| = \sqrt{\langle x, x \rangle} = \sqrt{\sum_{i=1}^{N} x_i^2} \quad \forall x \in \RR^N.\]

This makes \(\RR^N\) a normed linear space.

The angle \(\theta\) between two vectors is given by:

\[\theta = \cos^{-1} \frac{ \langle x, y \rangle }{\| x \| \| y \|}\]

Distance ¶

Distance between two vectors is defined as:

\[d(x,y) = \| x - y \| = \sqrt{\sum_{i=1}^{N} (x_i - y_i)^2}\]

This distance function is known as Euclidean metric.

This makes \(\RR^N\) a metric space.

\(\ell_p\) norms ¶

In addition to standard Euclidean norm, we define a family of norms indexed by \(p \in [1, \infty]\) known as \(l_p\) norms over \(\RR^N\).

Definition

\(\ell_p\) norm is defined as:

(2)¶\[\begin{split} \| x \|_p = \begin{cases} \left ( \sum_{i=1}^{N} | x |_i^p \right ) ^ {\frac{1}{p}} & p \in [1, \infty)\\ \underset{1 \leq i \leq N}{\max} |x_i| & p = \infty \end{cases}\end{split}\]

\(\ell_2\) norm ¶

As we can see from definition, \(\ell_2\) norm is same as Euclidean norm. So we have:

\[\| x \| = \| x \|_2\]

\(\ell_1\) norm ¶

From above definition we have

\[\|x\|_1 = \sum_{i=1}^N |x_i|= |x_1| + |x_2| + \dots + | x_N|\]

We use norms as a measure of strength of a signal or size of an error. Different norms signify different aspects of the signal.

Quasi-norms ¶

In some cases it is useful to extend the notion of \(\ell_p\) norms to the case where \(0 < p < 1\).

In such cases norm as defined in (2) doesn’t satisfy triangle inequality, hence it is not a proper norm function. We call such functions as quasi-norms.

\(\ell_0\)-“norm”¶

Of specific mention is \(\ell_0\)-“norm”. It isn’t even a quasi-norm. Note the use of quotes around the word norm to distinguish \(\ell_0\)-“norm” from usual norms.

Definition

\(\ell_0\)-“norm” is defined as:

\[\| x \|_0 = | \supp(x) |\]

where \(\supp(x) = \{ i : x_i \neq 0\}\) denotes the support of \(x\).

Note that \(\| x \|_0\) defined above doesn’t follow the definition in (2).

Yet we can show that:

\[\lim_{p\to 0} \| x \|_p^p = | \supp(x) |\]

which justifies the notation.