12. Wavelet scattering transform

The wavelet scattering transform is a new technique for signal processing based on networks of wavelet convolutions. It was pioneered by Stéphane Mallat in Paris in the early 2010s (Mallat, 2012). The technique can be applied to 1D signals like time series, 2D signals like images, or higher dimensional signals. In the past decade, it has found many applications in physics, music, computer vision, and finance.

Seminal references include:

We refer you to these references for technical details. Here we will focus on the intuition and in problem class you will implement a classifier using the scattering transform. In the remainder of this lecture we will refer to the wavelet scattering transform as the scattering transform.

12.1. Limitation of non-localized kernels

The foundation of Fourier analysis are sinusoidal waves (the kernel of the Fourier transform). Pure sinusoidal waves are infinite in extent and thus not localized (in time).

The fourier transform of a time-dependent signal is thus a function of frequency, which has the same value at all times. As you can imagine, a lot of information is lost in this representation.

To preserve some amount of time-locality, we can apply a Gaussian window to the signal before taking the Fourier transform. This leads to the short-time Fourier transform:

\[STFT[f](t, \omega) = \int f(t^\prime) g_T(t^\prime - t) e^{-2\pi i \omega (t-t^\prime)} dt^\prime\]

where \(g_T\) is a window function localized over a time interval of length \(T\). For instance, a Gaussian window function is given by:

\[g_T(t) = \frac{1}{\sqrt{2\pi T^2}} e^{-t^2/2T^2}\]

The result of the STFT is now a function of both time and frequency.

12.2. Wavelets

Instead of chosing a window function defined over a fixed length \(T\), it is useful to consider a family of window functions indexed by a parameter. We trade \(T\) for \(Q/\omega\) and consider the a Gaussian window function together with the oscillatory part. This gives rise to the Morlet wavelet (or Gabor wavelet), here with \(Q=1\):

\[\psi_{\omega}(t) = \frac{1}{\sqrt{2\pi\omega^2}} e^{-\omega^2 t^2/2} e^{-2\pi i \omega t}.\]

For a general value of \(Q\), the wavelet is denoted \(\psi_{\omega, Q}\).

In wavelet analysis, \(Q\) is chosen and kept fixed. It is called the quality factor which eventually controls the frequency resolution, or equivalently the number of frequencies per octave.

A constant-Q filter bank is constructed by considering all wavelets \(\psi_{\omega, Q}\) with \(\omega=2^{-j/Q}\omega_0\) for \(j=0, 1, 2, \ldots\), and \(\omega_0\) a central frequency that is also fixed (and is not important, as all what matters are the frequency ratios).

The wavelets in the filter bank are localized in time and also in frequency.

12.3. Wavelet transform

The wavelet transform (or coefficients) of a signal \(f\) is given by the convolution of \(f\) with all the wavelets in the constant-Q filter bank:

\[Wf(t,\omega) = \int f(t^\prime) \psi_{\omega, Q}(t^\prime - t) dt^\prime.\]

The modulus of the wavelet transform, computed for all wavelets in the constant-Q filter bank, and aggregated in a log-frequency axis, is called a scalogram.

The convolution is carried out in the Fourier domain, by a simple pointwise product (see convolution theorem).

Moreoever, the sum of the energy of all the coefficients is the same as the energy of the original signal. In other words, the wavelet transform preserves norm:

\[||Wf||^2=||f||^2.\]

This can be obtained from the Plancherel theorem, which states that the L^2 norm is preserved by the Fourier transform.

To be fully exhaustive, the first filter in the filter bank is in fact a low-pass filter (a simple Gaussian), and the following filters are band-pass filters (wavelets). The low-pass filter is generally denoted \(\phi_{2^J}\) with \(J\) a positive integer and \(2^J\) the averaging scale. This low-pass filter acts like a moving average over a time interval of length \(2^J\).

Thus, we can write the wavelet transform as \begin{align*} Wf & =\begin{pmatrix}f \star \phi_{2^J}(t) \\ f \star \psi_{\lambda}(t) \end{pmatrix}_{\lambda\leq 2^J} \end{align*} where \(\lambda = 2^{-j/Q}\) for \(j\in \mathbb{Z}\). The first coefficient is an average and the following coefficients contain the high-frequency information.

As we will see, the high-frequency part oscillates and we can kill the oscillations by taking the modulus. We adopt the notation:

\begin{align*} |W|f & =\begin{pmatrix}f \star \phi_{2^J}(t) \\ |f \star \psi_{\lambda}(t)| \end{pmatrix}_{\lambda\leq 2^J} \end{align*}

12.4. Building translation-invariant features

The standard approach to build a translation-invariant feature is to take the average. For instance, with \(J=\infty\), the coefficient

\[f \star \phi_{2^J}\]

is the average of the signal (in Fourier space, we have multiplied the transform of the signal by a Dirac delta, i.e., it is the the zero-frequency). When we took this average, we lost all the high-frequency information as we eliminated those frequencies.

The information that we have lost then is the high-frequency information.

How can we capture the high-frequency information?

We can capture the high-frequency information as wavelet coefficients. How does the high-frequency information look like, for a fixed \(\lambda\)? It is localized in time and oscillates. Explicitly, it is:

\[f\star \psi_{\lambda}(t) = f\star \mathrm{Re}\left(\psi_{\lambda}(t)\right) + i f\star \mathrm{Im}\left(\psi_{\lambda}(t)\right).\]

If we take the average here, we have something that will give zero, because of the fast oscillations.

To extract more invariant features, we need to be non-linear. Take the modulus of the wavelet coefficients:

\[|f\star \psi_{\lambda}(t)| = \sqrt{\left(f\star \mathrm{Re}\left(\psi_{\lambda}(t)\right)\right)^2 + \left(f\star \mathrm{Im}\left(\psi_{\lambda}(t)\right)\right)^2}.\]

This kills the oscillations and we are left with an envelope. However, the envelope is still not translation invariant, as if we shift the signal in time, the envelope will also shift. How can we make it translation invariant? Take the average. Let’s average:

\[|f\star \psi_{\lambda}| \star \phi_{2^J}\]

This gives us a translation-invariant feature (for translation less than \(2^J\)). More specifically, for each coefficient indexed by \(\lambda\), we end-up with \(N/2^J\) features. For instance, if we have \(N=1024\) time samples and \(J=8\), we extract \(4\) features per \(\lambda\).

But in doing so, we have lost the high-frequency information stored in the envelope.

12.5. Convolution cascade

How can we recover the high-frequency information stored in the envelope of the first wavelet coefficients? We can take a convolution with a second wavelet \(\psi_{\mu}\), namely

\[||f \star \psi_{\lambda}| \star \psi_{\mu}|\]

after also taking the modulus to kill the oscillations. We store these two previous results inside what we calle the second wavelet transform modulus:

\[\begin{split}\begin{align*} |W||f\star \psi_1| & =\begin{pmatrix}|f\star \psi_{\lambda}|\star \phi_{2^J}(t) \\ ||f \star \psi_{\lambda}| \star \psi_{\mu}(t) | \end{pmatrix} \end{align*}\end{split}\]

The sequence of operations that we are performing here corresponds to a convolution network with “linear filter, modulus, linear filter, modulus”. (The linear filter is the convolution with a wavelet).

12.6. Scattering transform coefficients

The second wavelet transform computes the wavelet transform of the envelope of the first. It corresponds to sending new waves through the envelope which makes its different parts interact, building interferences, yielding a second-order scattering coefficient that has memory of the geometry of the structure of the signal.

The scattering transform coefficients are the averaged parts of the wavelet transform moduli. They are defined up to third order as:

\[\begin{split}\begin{align*} S_Jf^{(0)}& = f \star \phi_{2^J}\\ S_Jf^{(1)}(\lambda) & = |f \star \psi_{\lambda}| \star \phi_{2^J}\\ S_Jf^{(2)}(\lambda,\mu) & = ||f \star \psi_{\lambda}| \star \psi_{\mu}| \star \phi_{2^J} \end{align*}\end{split}\]

The wavelet scattering transform (WST) is the object \(S_J f = [S_J f^{(0)}, S_J f^{(1)}, S_J f^{(2)}]\).

12.7. Applications

The WST takes us from a high-dimensional 1D signal (here by high-dimensional we mean that the number of time samples \(N\) is large) to a low-dimensional feature space (here by low-dimensional we mean that the number of features is much smaller than \(N\)). The feature space is made of scalograms. The features are the WST coefficients and they constitute an efficient, low-variance, low-dimensionality statistical description of the signal. This is ideal for classification tasks and for generative modelling.

The WST coefficients depend on high-order moments of \(x\), mainly of order up to \(2^m\) for the \(m^\text{th}\) (See Bruna & Mallat). We therefore expect the \(m=2\) coefficients to allow to distinguish fields that have the same second order moments (i.e. power spectra), but different higher order moments. For this reason, the WST can classify textures very efficiently.

Applications range from physics, finance, music, image recognition (e.g., medical imaging), and more.