Math for Deep Learning: arg min

arg min f(x) = the value x with the minimum value of f(x)

這個arg min f(x)的意思是，當f(x)是最小值的時候，x的數值

e.g. 以下的例子

f(0) = 3

f(0.9) = 2.1

f(1) = 2

f(1.1) = 2.5

f(2) = 5

f(3) = 46

因為最小的f(x) ＝ 2
則arg min f(x) = 1

Reference:

Explanation on arg min

Jacobian Matrix 雅可比矩陣

Jacobian Matrix 雅可比矩陣

The matrix that arranges the first-order partial derivatives of a function of a vector.

vector y is a function f of vector x:

vector y = f(vector x)
vector x = [x₁, x₂, ..., x_n]
vector y = [y₁, y₂, ..., y_m]

Jacobian matrix:

J = [∂f/x₁, ∂f/x₂, ..., ∂f/x_n]
  = [∂y₁/x₁, ∂y₁/x₂, ..., ∂y₁/x_n
      ∂y₂/x₁, ∂y₂/x₂, ..., ∂y₂/x_{n
      .....}
     ∂y_m/x₁, ∂y_m/x₂, ..., ∂y_m/x_n]

(The above figure of formula is from Wikipedia: Jacobian matrix and determinant)

References:

Jacobian Matrix (Wikipedia) 雅可比矩陣 (維基百科)
Autograd: Automatic Differentiation (PyTorch Official Tutorial)

Matlab: Singular and Nonsingular Matrices 奇異矩陣/非奇異矩陣

inverse matrix 逆矩陣/反矩陣

Nonsingular

If determinant ≠ 0 => i.e. the inverse matrix exists.
=> the matrix is invertible/non-singular.

invertible matrix 可逆矩陣
nonsingular matrix 非奇異矩陣/非特異矩陣

nondegenerate matrix 非退化矩陣

AB = BA = I

B = A^-1 = adj(A)/det(A)

where
adj(A) = adjoint matrix 伴隨矩陣 of A
det(A) = determinant 行列式 of A

>> a = [1 1 ; 2 1]

a =

     1     1
     2     1

>> det(a)

ans =

    -1

>> b = inv(a)

b =

    -1     1
     2    -1

>> a*b

ans =

     1     0
     0     1

>> b*a

ans =

     1     0
     0     1

Singular

If determinant = 0 => the matrix is invertible.

non-invertible matrix 不可逆矩陣

singular matrix 奇異矩陣/特異矩陣
degenerate matrix 退化矩陣

>> a = [2 3;1 1.5]

a =

    2.0000    3.0000
    1.0000    1.5000

>> det(a)

ans =

     0

>> b = inv(a)
Warning: Matrix is singular to working precision. 

b =

   Inf   Inf
   Inf   Inf

>>

Example:

References

奇異矩陣 singular matrix (國家教育研究院雙語詞彙、學術名詞暨辭書資訊網)
Invertible matrix (Wikipedia)

Correlation Matrix 相關矩陣

correlation matrix 相關矩陣
一個穩態離散時間隨機過程可以一個時間序列來表示(M×1)，而當我們把這個隨機時間序列乘以它的赫密特轉置矩陣(1×M)，所得到的期望值(因為時間序列是隨機的)即是這個時間序列的相關矩陣(M×M)。

R = E[ u(n) u^H(n) ]

R: correlation matrix
u(n) : stochastic process
u^H(n) : Hermitian transpose
E: expectation

Hermitian transpose 赫密特轉置矩陣
將一個矩陣轉置並取共軛複數(complex conjugate)

在Matlab中，可直接以'符號求得Hermitian transpose，或使用ctranspose function

>> a = [1 1+j;2-j -2]

a =

   1.0000 + 0.0000i   1.0000 + 1.0000i
   2.0000 - 1.0000i  -2.0000 + 0.0000i

>> a'

ans =

   1.0000 + 0.0000i   2.0000 + 1.0000i
   1.0000 - 1.0000i  -2.0000 + 0.0000i


Hermitian matrix/self-adjoint matrix 埃爾米特矩陣/厄米特矩陣/自伴隨矩陣

共軛對稱的方塊矩陣，即 A = (A^T)^*，則A的Hermitian transpose等於A自己，即 A^H = A

Example:

>> a = [1 3+j; 3-j -2]

a =

   1.0000 + 0.0000i   3.0000 + 1.0000i
   3.0000 - 1.0000i  -2.0000 + 0.0000i

>> a'

ans =

   1.0000 + 0.0000i   3.0000 + 1.0000i
   3.0000 - 1.0000i  -2.0000 + 0.0000i

>> isequal(a,a')

ans =

  logical

   1

>> 

References:

Haykin, S. S. (2014). Adaptive filter theory. 5th edition, Pearson Education. pp 52-56.
Conjugate transpose (Wikipedia)
Hermitian matrix (Wikipedia) 埃爾米特矩陣(維基百科)

Matlab: Sinc function

The code below shows how to use the sinc() function of Matlab's Signal Processing Toolbox:

x = (-6:1/100:6);
y1 = sinc(x);
plot(x,y1);
y2 = sinc(2*x);
plot(x,y2);
xlabel('x');ylabel('sinc'); title('Sinc Function')
legend('sinx(x)', 'sinc(2x)')

Result:

Note:
The sinc() function here is defined as

    sinc(x) = sin(πx)/πx (normalized sinc function)

References:

sinc - sinc function (MathWorks)
MATLAB sinc issue (StackOverflow)

Matlab: Find the arguments of maxima/minima (argmax/argmin) of a vector

The argument of maximum means the index at which the vector/function values is maximized.

Example

>> a = [3 2 1 -2 7]

a =
     3     2     1    -2     7

>> [max_value, argmax] = max(a)

max_value =
     7

argmax =
     5

>> [min_value, argmin] = min(a)

min_value =
    -2

argmin =
     4

>> 

References:

Arg max (Wikipedia)
how do i find argmax? (MathWorks)

Mathematics for Machine Learning 機器學習的數學

Siraj Raval的影片介紹，機器學習主要有以下幾種的數學：

Calculus 微積分 - 最佳化

Linear Algebra 線性代數 - 實現演算法

Probability 機率 - 預估結果

Statistics 統計 - 找出目標

----

參考資料

Mathematics of Machine Learning (Siraj Raval)
Logistic regression (Wikipedia) 邏輯迴歸 (維基百科)

Identity Matrix 單位矩陣

Identity Matrix / Unit Matrix 單位矩陣/恆等矩陣

Example:

I =
[ 1 0 ]
[ 0 1 ]

I =
[ 1 0 0 ]
[ 0 1 0 ]
[ 0 0 1 ]

I =
[ 1 0 0 0 ]
[ 0 1 0 0 ]
[ 0 0 1 0 ]
[ 0 0 0 1 ]

Matlab Example:

>> I = eye(4)

I =

     1     0     0     0
     0     1     0     0
     0     0     1     0
     0     0     0     1

>>

For an NxN identity matrix I and a NxN matrix A,

AI = IA = A

For an NxN identity matrix I and a Nx1 matrix B or a 1xN matrix C,

IB = B
CI = C

Reference:

Identity Matrix (Wikipedia/維基百科)

Matlab: Laplace Transform

The examples below demonstrate the Laplace transforms and inverse Laplace transforms using Matlab symbolic number or variable.

Laplace Transform

L{1} = 1/s

>> x = sym(1);
>> laplace(x)

ans =

1/s

L{t} = 1/s²

>> syms t
>> laplace(t)

ans =

1/s^2

Inverse Laplace Transform

L^-1{1/s} = 1

>> syms s
>> ilaplace(1/s)

ans =

1

L^-1{1/(s-a)} = e^at where a = 2

>> ilaplace(1/(s-2))

ans =

exp(2*t)

L^-1{a²/(s²+a²)} = sin(at) where a = 1

>> ilaplace(1/(s^2+1))

ans =

sin(t)

Reference

Zill, D., Wright, W. S., & Cullen, M. R. (2011). Advanced engineering mathematics. 4th Edition. Jones & Bartlett Learning. p 197-198.

laplace / ilaplace - MathWorks Documentation

Mathematical symbols 數學符號

∀ - for all

Reference

List of mathematical symbols (Wikipedia)

Tensor 張量

Array of numbers that may have
- scalar 純量
- vector 向量
- matrix 矩陣
- more dimensions.

Refernces

Lecture slides for Chapter 2 of Deep Learning, Ian Goodfellow
張量 (Tensor) - 金門大學陳鍾誠的網站

Vectors 向量

dot product / scalar product / inner product / projection product 點積/內積/純量積 a·b
A·B = |A| |B| cos(θ)
A_B = |A| cos(θ)

For vectors a = (a₁, a₂, ..., a_n) and b = (b₁, b₂, ..., b_n) ,

a·b = a₁b₁ + a₂b₂ + ... + a_nb_n

For example, vectors a = (1, 2, 3) and b = (0, 4, 5),

a·b = 1×0 + 2×4 + 3×5 = 23

Matlab code:

>> a = [1 2 3];
>> b = [0 4 5];
>> dot(a,b)

ans =

    23

cross product/vector product 點積/向量積 a×b

參考資料

內積與外積

Partial Fraction 部分分式

Partial fraction decomposition may be required for dealing with z-transform.

partial fraction decomposition 部分分式分解
partial fraction expansion 部分分式展開

Solution:

Let X/( 1 )( 2 ) = A/( 1 ) + B/( 2 )

Multiply ( 1 ) and ( 2 ) at both sides and get

X = A ( 2 ) + B ( 1 )

Let ( 1 ) = 0 to get A:
A = X / ( 2 ) | _{( 1 ) = 0}

Let ( 2 ) = 0 to get B:
B = X / ( 1 ) | _{( 2 ) = 0}

Base and Exponent 底數與指數

For xⁿ

中文讀法：x的n次方
英文讀法：x to the power of n

where

x:
base 底數

n:
index/exponent 指數
power 次方

Eigendecomposition, Eigenvalue, Eigenvector, Eigenfunction 特徵分解、特徵值、特徵向量、特徵函數/固有值、固有向量、固有函數

Eigendecomposition 特徵分解
Spectral decomposition 譜分解

scalar 純量
vector 向量

For n×n matrix A,

if Ax = λx

where
λ is a scalar.

x is a non-zero vector with dimension N.

Then

Av = λv
Eigenvalue 特徵值/固有值 λ
Eigenvector 特徵向量/固有向量 v

Af = λf
Eigenfunction 特徵函數/固有函數 f

For an LTI system, an input signal x[n] which is an eigenfunction of the system such as x[n] = e^jωn appears at the output of the system with the eigenvalue H(e^jω).

Magnitude of Exponential Function e^jθ 如何求e^jθ的大小/絕對值

How to find the magnitude of exponential function e^iθ? Or |e^iθ| = ?

According to Euler's formula,

e^iθ = cosθ + isinθ

To find the magnitude of |e^iθ|,

|e^iθ| = SQRT(e^iθ) = SQRT(cos²θ + isin²θ) = SQRT(1) = 1

Therefore,

the magnitude of e^iθ is 1.

In other words, the magnitude of complex exponential is 1.

Related Information:

Euler's formula 歐拉公式
Why is the magnitude of e^(jwt) equal to 1?
Absolute value of complex exponential

Arithmetic/Geometric Sequence, Series, and Progression 等差/等比的數列/級數

等差數列/算數數列 arithmetic sequence/arithmetic progression 1,3,5,7...
等差級數/算數級數 arithmetic series 1+3+5+7+...
等差級數和：

n(a₁ + a_n)/2

1+3+5+7+9 = 5×(1+9)/2 = 25

等比數列/幾合數列 geometric sequence/geometric progression 1,2,4,8...
等比級數/幾合級數 geometric series 1+2+4+8+...

等比級數和：

a(1-rⁿ)/(1-r) = a(rⁿ-1)/(r-1)

1+2+4+8+16 = 1×(1-2⁵)/(1-2) = 1×(2⁵-1)/(2-1) = 1×(32-1)/(2-1) = 31

公差 common difference
公比 common ratio

參考資料

https://en.wikipedia.org/wiki/Arithmetic_progression
https://en.wikipedia.org/wiki/Geometric_progression
國家教育研究院雙語詞彙、學術名詞暨辭書資訊網

Negative and Reciprocal Inequalities 負數及倒數的不等式運算規則

要理解負數及倒數的不等式運算規則
其實很簡單：

Negative Number 負數：

if a < b, then -a > -b

e.g. 3 < 5 ==> -3 > -5

Reciprocal/Multiplicative Inverse 倒數：

if a < b, then 1/a > 1/b

e.g. 3 < 5 ==> 1/3 > 1/5

Reference 參考資料：

Properties of Inequalities

convolution and DSP

convolution 摺積/捲積/疊積/旋積
convolution theorem 捲積定理

input: x[n]
system: h[n]
output: y[n]


convolution in time domain = multiplication in frequency domain

Time Domain:

y(n) = h(n) * x(n) = Σ h(k)x(n-k) where k ∈ Z (Z: integers 整數 ... -3, -2, -1, 0, 1, 2, 3, ...)

Frequency Domain:

Y(k) = H(k)X(k)

相關資料

Convolution (Wikipedia) / 摺積 (維基百科)
Video: Introduction to the convolution (Khan Academy 可汗學院)
Meaning of R, Q, Z, and N characters in math 數學中R, Q, Z, N的意義

Meaning of R, Q, Z, and N characters in math 數學中R, Q, Z, N的意義

R - real numbers 實數 -∞ ~ ∞, no imaginary part i 不含虛部 i

Q - rational numbers 有理數 may be represented as a ratio of p/q, where q ≠ 0.

Z - integers 整數 ... -3, -2, -1, 0, 1, 2, 3, ...

N - natural numbers 自然數 1, 2, 3, 4, ...

R 包含了 Q
Q 包含了 Z
Z 包含了 N

Figure of R, Q, Z, and N relationship

Example:

point sampling

x[n] = x(nT), n ∈ Z
n 屬於整數

參考資料

https://en.wikipedia.org/wiki/Rational_number#/media/File:Number-systems.svg