Matlab: Parseval's theorem

Parseval's theorem states that for N discrete points of signal,

Total Energy ∑ |x[n]|² = 1/N × ∑ |X[k]|²

where X[k] is the k^th point of the discrete Fourier transform of x[n].

Let's check this with Matlab:

load mtlb; %Load "Matlab" sound example

N = 128;

y = mtlb(1:N); %Read the 1st 128 points only

%Check Parseval's theorem

%y^2

y_2 = y.^2;

%Energy

energy = sum(y_2)

%Frequency domain

y_fft = fft(y); %Complex double

%|y_fft|^2

y_fft_2 = abs(y_fft).^2;

y_fft_2_check = y_fft.*conj(y_fft); %Should be identical to y_fft_2

energy_fft = sum(y_fft_2)/N

energy_fft_check = sum(y_fft_2_check)/N

Result:

Reference:

energy =

    2.5595


energy_fft =

    2.5595


energy_fft_check =

    2.5595

>> 

Parseval's theorem (Wikipedia)

Matlab: Paste figure into Microsoft Word or PowerPoint files

If Matlab figures are pasted into Microsoft Office applications such as Word and PowerPoint, the file becomes slow and the mouse icon is turning/loading.

I have figured out the following solution:

1. Select copy figure

2. Paste into an empty Word file.

3. Select the figure just pasted in the Word file and copy it.

4. Past into Gimp. You should NOT see this warning:

(If you see this error message, you probably directly pasted Matlab figure into Gimp. You cannot directly do this, so steps 2 and 3 are essential. The transparency can be preserved in this way.)

5. Select all in the image layer of Gimp and copy it.

6. Paste into the target Word/PowerPoint file.

Machine Learning - Batch Size and Epoch

Batch Size 批大小

The number of training examples present in a single batch.
一批次訓練中的樣本數，因無法將完整dataset送入整個神經網路訓練，故將dataset分割為數個批次(divide the dataset into batches)，例如batch_size = 100，即一個批次中有100個樣本。

Epoch

Passing an entire dataset forward and backward through the neural network once.
一個資料集(dataset)完整forward和backward通過神經網络的過程

Iteration 迭代

One iteration is the number of batches needed to complete one epoch.
做完一個epoch的訓練需要數個batch的樣本，稱為一個iteration。

Example

對於一個3000個訓練樣本的資料集(Dataset)。
若將這3000個樣本以batch_size=500分割，則完成一個epoch需要6個iterations。
做一個iteration，需以一批500個樣本進行訓練。

References 參考資料：
神經網路中Epoch、Iteration、Batchsize相關理解和說明
深度學習中的 epoch iteration batch-size
Epoch vs Batch Size vs Iterations

Jacobian Matrix 雅可比矩陣

Jacobian Matrix 雅可比矩陣

The matrix that arranges the first-order partial derivatives of a function of a vector.

vector y is a function f of vector x:

vector y = f(vector x)
vector x = [x₁, x₂, ..., x_n]
vector y = [y₁, y₂, ..., y_m]

Jacobian matrix:

J = [∂f/x₁, ∂f/x₂, ..., ∂f/x_n]
  = [∂y₁/x₁, ∂y₁/x₂, ..., ∂y₁/x_n
      ∂y₂/x₁, ∂y₂/x₂, ..., ∂y₂/x_{n
      .....}
     ∂y_m/x₁, ∂y_m/x₂, ..., ∂y_m/x_n]

(The above figure of formula is from Wikipedia: Jacobian matrix and determinant)

References:

Jacobian Matrix (Wikipedia) 雅可比矩陣 (維基百科)
Autograd: Automatic Differentiation (PyTorch Official Tutorial)

Matlab: Singular and Nonsingular Matrices 奇異矩陣/非奇異矩陣

inverse matrix 逆矩陣/反矩陣

Nonsingular

If determinant ≠ 0 => i.e. the inverse matrix exists.
=> the matrix is invertible/non-singular.

invertible matrix 可逆矩陣
nonsingular matrix 非奇異矩陣/非特異矩陣

nondegenerate matrix 非退化矩陣

AB = BA = I

B = A^-1 = adj(A)/det(A)

where
adj(A) = adjoint matrix 伴隨矩陣 of A
det(A) = determinant 行列式 of A

>> a = [1 1 ; 2 1]

a =

     1     1
     2     1

>> det(a)

ans =

    -1

>> b = inv(a)

b =

    -1     1
     2    -1

>> a*b

ans =

     1     0
     0     1

>> b*a

ans =

     1     0
     0     1

Singular

If determinant = 0 => the matrix is invertible.

non-invertible matrix 不可逆矩陣

singular matrix 奇異矩陣/特異矩陣
degenerate matrix 退化矩陣

>> a = [2 3;1 1.5]

a =

    2.0000    3.0000
    1.0000    1.5000

>> det(a)

ans =

     0

>> b = inv(a)
Warning: Matrix is singular to working precision. 

b =

   Inf   Inf
   Inf   Inf

>>

Example:

References

奇異矩陣 singular matrix (國家教育研究院雙語詞彙、學術名詞暨辭書資訊網)
Invertible matrix (Wikipedia)

Correlation Matrix 相關矩陣

correlation matrix 相關矩陣
一個穩態離散時間隨機過程可以一個時間序列來表示(M×1)，而當我們把這個隨機時間序列乘以它的赫密特轉置矩陣(1×M)，所得到的期望值(因為時間序列是隨機的)即是這個時間序列的相關矩陣(M×M)。

R = E[ u(n) u^H(n) ]

R: correlation matrix
u(n) : stochastic process
u^H(n) : Hermitian transpose
E: expectation

Hermitian transpose 赫密特轉置矩陣
將一個矩陣轉置並取共軛複數(complex conjugate)

在Matlab中，可直接以'符號求得Hermitian transpose，或使用ctranspose function

>> a = [1 1+j;2-j -2]

a =

   1.0000 + 0.0000i   1.0000 + 1.0000i
   2.0000 - 1.0000i  -2.0000 + 0.0000i

>> a'

ans =

   1.0000 + 0.0000i   2.0000 + 1.0000i
   1.0000 - 1.0000i  -2.0000 + 0.0000i


Hermitian matrix/self-adjoint matrix 埃爾米特矩陣/厄米特矩陣/自伴隨矩陣

共軛對稱的方塊矩陣，即 A = (A^T)^*，則A的Hermitian transpose等於A自己，即 A^H = A

Example:

>> a = [1 3+j; 3-j -2]

a =

   1.0000 + 0.0000i   3.0000 + 1.0000i
   3.0000 - 1.0000i  -2.0000 + 0.0000i

>> a'

ans =

   1.0000 + 0.0000i   3.0000 + 1.0000i
   3.0000 - 1.0000i  -2.0000 + 0.0000i

>> isequal(a,a')

ans =

  logical

   1

>> 

References:

Haykin, S. S. (2014). Adaptive filter theory. 5th edition, Pearson Education. pp 52-56.
Conjugate transpose (Wikipedia)
Hermitian matrix (Wikipedia) 埃爾米特矩陣(維基百科)

Adaptive Equalizer 自適應性等化器

在通訊系統中，例如對於電話數據機(modem)的資料傳輸，在低位元率(low bit rate)時，通常比較沒有符號間干擾/符際干擾(Intersymbol Interference, ISI)的問題。

但是當傳輸位元率提高時，例如超過2400 bits時，ISI的問題會變嚴重，這時就需要使用自適應性頻道等化器(Adaptive Channel Equalizer)來克服通道失真(Channel Distortion)的問題。

Adaptive Equalizer當見的實現方式是使用自適應性有限脈衝響應濾波器(Adaptive FIR Filter)，其係數採用LMS演算法(Least-Mean-Square Algorithm, 最小均方演算法)進行調整，以克服通道失真。LMS演算法是一種隨機梯度演算法(Stochastic Gradient Algorithm)。

通了使用LMS，Adaptive Equalizer也可採用RLS演算法(Recursive Least-Squares Algorithm, 遞迴最小平方演算法)。

參考資料：

Adaptive equalizer (Wikipedia)
Stochastic gradient descent (Wikipedia) (隨機梯度下降)
Least mean squares filter (Wikipedia) (屬於一種Stochastic Gradient Algorithm)
Recursive least squares filter (Wikipedia)

Haykin, S. S. (2014). Adaptive filter theory. 5th edition, Pearson Education. pp 40-43, 468-471.

Ingle, V. K., & Proakis, G. J. (2014). Essentials of Digital Signal Processing using MATLAB®. CENGAGE Learning, pp 603-606.