Matlab: Parseval's theorem

Parseval's theorem states that for N discrete points of signal,

Total Energy ∑ |x[n]|² = 1/N × ∑ |X[k]|²

where X[k] is the k^th point of the discrete Fourier transform of x[n].

Let's check this with Matlab:

load mtlb; %Load "Matlab" sound example

N = 128;

y = mtlb(1:N); %Read the 1st 128 points only

%Check Parseval's theorem

%y^2

y_2 = y.^2;

%Energy

energy = sum(y_2)

%Frequency domain

y_fft = fft(y); %Complex double

%|y_fft|^2

y_fft_2 = abs(y_fft).^2;

y_fft_2_check = y_fft.*conj(y_fft); %Should be identical to y_fft_2

energy_fft = sum(y_fft_2)/N

energy_fft_check = sum(y_fft_2_check)/N

Result:

Reference:

energy =

    2.5595


energy_fft =

    2.5595


energy_fft_check =

    2.5595

>> 

Parseval's theorem (Wikipedia)

Matlab: Paste figure into Microsoft Word or PowerPoint files

If Matlab figures are pasted into Microsoft Office applications such as Word and PowerPoint, the file becomes slow and the mouse icon is turning/loading.

I have figured out the following solution:

1. Select copy figure

2. Paste into an empty Word file.

3. Select the figure just pasted in the Word file and copy it.

4. Past into Gimp. You should NOT see this warning:

(If you see this error message, you probably directly pasted Matlab figure into Gimp. You cannot directly do this, so steps 2 and 3 are essential. The transparency can be preserved in this way.)

5. Select all in the image layer of Gimp and copy it.

6. Paste into the target Word/PowerPoint file.

Machine Learning - Batch Size and Epoch

Batch Size 批大小

The number of training examples present in a single batch.
一批次訓練中的樣本數，因無法將完整dataset送入整個神經網路訓練，故將dataset分割為數個批次(divide the dataset into batches)，例如batch_size = 100，即一個批次中有100個樣本。

Epoch

Passing an entire dataset forward and backward through the neural network once.
一個資料集(dataset)完整forward和backward通過神經網络的過程

Iteration 迭代

One iteration is the number of batches needed to complete one epoch.
做完一個epoch的訓練需要數個batch的樣本，稱為一個iteration。

Example

對於一個3000個訓練樣本的資料集(Dataset)。
若將這3000個樣本以batch_size=500分割，則完成一個epoch需要6個iterations。
做一個iteration，需以一批500個樣本進行訓練。

References 參考資料：
神經網路中Epoch、Iteration、Batchsize相關理解和說明
深度學習中的 epoch iteration batch-size
Epoch vs Batch Size vs Iterations

Jacobian Matrix 雅可比矩陣

Jacobian Matrix 雅可比矩陣

The matrix that arranges the first-order partial derivatives of a function of a vector.

vector y is a function f of vector x:

vector y = f(vector x)
vector x = [x₁, x₂, ..., x_n]
vector y = [y₁, y₂, ..., y_m]

Jacobian matrix:

J = [∂f/x₁, ∂f/x₂, ..., ∂f/x_n]
  = [∂y₁/x₁, ∂y₁/x₂, ..., ∂y₁/x_n
      ∂y₂/x₁, ∂y₂/x₂, ..., ∂y₂/x_{n
      .....}
     ∂y_m/x₁, ∂y_m/x₂, ..., ∂y_m/x_n]

(The above figure of formula is from Wikipedia: Jacobian matrix and determinant)

References:

Jacobian Matrix (Wikipedia) 雅可比矩陣 (維基百科)
Autograd: Automatic Differentiation (PyTorch Official Tutorial)

Matlab: Singular and Nonsingular Matrices 奇異矩陣/非奇異矩陣

inverse matrix 逆矩陣/反矩陣

Nonsingular

If determinant ≠ 0 => i.e. the inverse matrix exists.
=> the matrix is invertible/non-singular.

invertible matrix 可逆矩陣
nonsingular matrix 非奇異矩陣/非特異矩陣

nondegenerate matrix 非退化矩陣

AB = BA = I

B = A^-1 = adj(A)/det(A)

where
adj(A) = adjoint matrix 伴隨矩陣 of A
det(A) = determinant 行列式 of A

>> a = [1 1 ; 2 1]

a =

     1     1
     2     1

>> det(a)

ans =

    -1

>> b = inv(a)

b =

    -1     1
     2    -1

>> a*b

ans =

     1     0
     0     1

>> b*a

ans =

     1     0
     0     1

Singular

If determinant = 0 => the matrix is invertible.

non-invertible matrix 不可逆矩陣

singular matrix 奇異矩陣/特異矩陣
degenerate matrix 退化矩陣

>> a = [2 3;1 1.5]

a =

    2.0000    3.0000
    1.0000    1.5000

>> det(a)

ans =

     0

>> b = inv(a)
Warning: Matrix is singular to working precision. 

b =

   Inf   Inf
   Inf   Inf

>>

Example:

References

奇異矩陣 singular matrix (國家教育研究院雙語詞彙、學術名詞暨辭書資訊網)
Invertible matrix (Wikipedia)

Correlation Matrix 相關矩陣

correlation matrix 相關矩陣
一個穩態離散時間隨機過程可以一個時間序列來表示(M×1)，而當我們把這個隨機時間序列乘以它的赫密特轉置矩陣(1×M)，所得到的期望值(因為時間序列是隨機的)即是這個時間序列的相關矩陣(M×M)。

R = E[ u(n) u^H(n) ]

R: correlation matrix
u(n) : stochastic process
u^H(n) : Hermitian transpose
E: expectation

Hermitian transpose 赫密特轉置矩陣
將一個矩陣轉置並取共軛複數(complex conjugate)

在Matlab中，可直接以'符號求得Hermitian transpose，或使用ctranspose function

>> a = [1 1+j;2-j -2]

a =

   1.0000 + 0.0000i   1.0000 + 1.0000i
   2.0000 - 1.0000i  -2.0000 + 0.0000i

>> a'

ans =

   1.0000 + 0.0000i   2.0000 + 1.0000i
   1.0000 - 1.0000i  -2.0000 + 0.0000i


Hermitian matrix/self-adjoint matrix 埃爾米特矩陣/厄米特矩陣/自伴隨矩陣

共軛對稱的方塊矩陣，即 A = (A^T)^*，則A的Hermitian transpose等於A自己，即 A^H = A

Example:

>> a = [1 3+j; 3-j -2]

a =

   1.0000 + 0.0000i   3.0000 + 1.0000i
   3.0000 - 1.0000i  -2.0000 + 0.0000i

>> a'

ans =

   1.0000 + 0.0000i   3.0000 + 1.0000i
   3.0000 - 1.0000i  -2.0000 + 0.0000i

>> isequal(a,a')

ans =

  logical

   1

>> 

References:

Haykin, S. S. (2014). Adaptive filter theory. 5th edition, Pearson Education. pp 52-56.
Conjugate transpose (Wikipedia)
Hermitian matrix (Wikipedia) 埃爾米特矩陣(維基百科)

Adaptive Equalizer 自適應性等化器

在通訊系統中，例如對於電話數據機(modem)的資料傳輸，在低位元率(low bit rate)時，通常比較沒有符號間干擾/符際干擾(Intersymbol Interference, ISI)的問題。

但是當傳輸位元率提高時，例如超過2400 bits時，ISI的問題會變嚴重，這時就需要使用自適應性頻道等化器(Adaptive Channel Equalizer)來克服通道失真(Channel Distortion)的問題。

Adaptive Equalizer當見的實現方式是使用自適應性有限脈衝響應濾波器(Adaptive FIR Filter)，其係數採用LMS演算法(Least-Mean-Square Algorithm, 最小均方演算法)進行調整，以克服通道失真。LMS演算法是一種隨機梯度演算法(Stochastic Gradient Algorithm)。

通了使用LMS，Adaptive Equalizer也可採用RLS演算法(Recursive Least-Squares Algorithm, 遞迴最小平方演算法)。

參考資料：

Adaptive equalizer (Wikipedia)
Stochastic gradient descent (Wikipedia) (隨機梯度下降)
Least mean squares filter (Wikipedia) (屬於一種Stochastic Gradient Algorithm)
Recursive least squares filter (Wikipedia)

Haykin, S. S. (2014). Adaptive filter theory. 5th edition, Pearson Education. pp 40-43, 468-471.

Ingle, V. K., & Proakis, G. J. (2014). Essentials of Digital Signal Processing using MATLAB®. CENGAGE Learning, pp 603-606.

Arduino: Button Detection with LED Control

1. Connect the button circuit according to the official Button page.

2. Connect the LED circuit to pin 13 according to the official Blink page or the circuit in the "Hello World with blinking LED" example.

3. Connect your Arduino to a computer via a USB cable.

4. Select Example -> 02.Digital -> Button.

5. Download the code. The LED should be turned on by Arduino when the button is pressed.

Related Information:

Arduino: Hello World! (StudyEECC)

Arduino: Hello World for WeMos D1 WIFI UNO(StudyEECC)
Arduino: Hello World with blinking LED (StudyEECC)

Arduino: Hello World with blinking LED

This tutorial shows how to display "Hello World" by Arduino and blink the built-in LED on the Arduino board or blink an external LED via wire connections.

Before working on this example, make sure you understand the pure Hello World example for Arduino.

Part A. Arduino built-in LED

1. Select File->Examples->01.Basics->Blink

2.  Modify the code as:

int count=0;
void setup() {
  Serial.begin(9600);
  // initialize digital pin LED_BUILTIN as an output.
  pinMode(LED_BUILTIN, OUTPUT);
}

// the loop function runs over and over again forever
void loop() {
  digitalWrite(LED_BUILTIN, HIGH);   // turn the LED on (HIGH is the voltage level)
  Serial.println("Hello, world!");
  delay(1000);                       // wait for a second
  digitalWrite(LED_BUILTIN, LOW);    // turn the LED off by making the voltage LOW
  Serial.println(count);
  delay(1000);                       // wait for a second
  count++;
}

3. Download the code and open the serial monitor

Result:

You should also see the LED on the Arduino board blinking.

Part B. External LED

1. Follow the figure shown in the official page for circuit connections from pin 13 to ground (GND) via an LED. Connect the longer leg / pin of the LED (called the anode) to the pin 13 with 5V power. The resistor value should be between 220 ohm and 1k ohm, but for me 2k ohm also works.

2. Modify the code as below:

int count=0;
int LED13=13; // For the external LED at pin 13
void setup() {
  Serial.begin(9600);
  // initialize digital pin LED_BUILTIN as an output.
  pinMode(LED_BUILTIN, OUTPUT);
  pinMode(LED13, OUTPUT);
}

// the loop function runs over and over again forever
void loop() {
  digitalWrite(LED_BUILTIN, HIGH);   // turn the LED on (HIGH is the voltage level)
  digitalWrite(LED13, HIGH);   // turn on the external LED at pin 13
  Serial.println("Hello, world!");
  delay(1000);                       // wait for a second
  digitalWrite(LED_BUILTIN, LOW);    // turn the LED off by making the voltage LOW
  digitalWrite(LED13, LOW);         // turn off the external LED at pin 13
  Serial.println(count);
  delay(1000);                       // wait for a second
  count++;
}

Result:

Reference:

Arduino: Hello, World! (StudyEECC)

Arduino: Hello World for WeMos D1 WIFI UNO(StudyEECC)
Arduino: Button Detection with LED Control (StudyEECC)

Arduino: Hello, World!

This is the first tutorial with Arduino. "Hello World" is usually the very first program to learn a new programming language with a simple command that prints out the "Hello, World!" message.

2. Connect the Arduino board to your computer via a USB cable. Note that in this case the power of the Arduino board is supplied by the 5V USB cable and it is not required to connect to an external DC adapter.

3. In the Arduino IDE window, type the following code:

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  Serial.println("Hello world, Arduino!");
}

void loop() {
  // put your main code here, to run repeatedly:
}

3. Select Tools -> your Arduino board such as the popular Arduino Uno.

4. Select Tools -> COM port.

5. Select the upload button and save the project.

6. Select Tools -> Serial monitor.

7. You should see the result if the serial monitor is properly set.

Reference:

Hello World
Arduino: Hello World with blinking LED (StudyEECC)

Arduino: Hello World for WeMos D1 WIFI UNO(StudyEECC)
Arduino: Button Detection with LED Control (StudyEECC)

Hello World with TensorFlow

After TensorFlow is installed, you may test the Hello World with commands in Python:

Type python to enter the Python environment:

python

Type the following commands

>>> import tensorflow as tf

>>> hello = tf.constant('Hello, World!')

>>> sess = tf.Session()

>>> sess.run(hello)

b'Hello, World!'

>>> 

Note that b'Hello, World!' represents byte literals.

If you want to print the message in a python file, use this command:

print(sess.run(hello))

Reference:

helloworld.py (GitHub:aymericdamien/TensorFlow-Examples)
What does the 'b' character do in front of a string literal? (Stack Overflow)

Matlab: FIR comb filter with various zero placement

The following Matlab code tests the effects of different number of zeros and checks the linear phase result.

a = 1;

b1 = [1 -1];

zplane(b1,a);
fvtool(b1,a);
LinPhase1 = islinphase(b1,a) %Check if linear phase

%Notch comb filter
b2 = [1 0 -1]; 
zplane(b2,a);
fvtool(b2,a);
LinPhase2 = islinphase(b2,a) %Check if linear phase

%Notch comb filter
b3 = [1 0 0 -1]; 
zplane(b3,a);
fvtool(b3,a);
LinPhase3 = islinphase(b3,a) %Check if linear phase

%Notch comb filter
b4 = [1 0 0 0 -1];
zplane(b4,a);
fvtool(b4,a);

%Notch comb filter
b5 = [1 0 0 0 0 -1];
zplane(b5,a);
fvtool(b5,a);
LinPhase5 = islinphase(b5,a) %Check if linear phase

Result:

LinPhase1 =

  logical
   1

LinPhase2 =

  logical
   1

LinPhase3 =

  logical
   1

LinPhase5 =

  logical
   1

b1 = [1 -1];

b2 = [1 0 -1]; 

b3 = [1 0 0 -1]; 

b4 = [1 0 0 0 -1]; 

 b5 = [1 0 0 0 -1]; 

Matlab: IIR Comb Notch or Peak Filter

IIR Comb Notch Filter

fs = 600;
fo = 100;
q = 35;
bw = (fo/(fs/2))/q;
[b,a] = iircomb(fs/fo,bw,'notch'); % Note type flag 'notch'
fvtool(b,a);
zplane(b,a);

Magnitude Response

Phase Response

Pole-Zero Diagram

IIR Comb Peak Filter

Replace the 'notch' string by 'peak':

[b,a] = iircomb(fs/fo,bw,'peak'); % Note type flag 'peak'

Magnitude Response

Phase Response

Pole-Zero Diagram

Reference

iircomb (MathWorks)