Deep Learning: threshold θ and bias b

The output y of a single-node neural network can be represented as:

y = a(Σw_ix_i - θ)

where a is the activation function, x_i is the i^th input, w_i is the i^th weight, and θ is the threshold.

When the summation of wighted input Σw_ix_i is less than θ, the neuron does not output. Hence we call θ the threshold, which is similar to the threshold potential in a physical neuron.

We may replace θ by -b and get

y = a(Σw_ix_i + b)

where b is called the bias parameter.

So b = - θ is a more generalized representation for the threshold.

References:

A Beginner’s Guide to Neural Networks: Part Two

Hinton Neural Networks課程筆記2b：第一代神經網路之感知機

深度學習的數學：用數學開啟深度學習的大門(博碩) p.12-16