ANN Building Blocks part 1
Biological Neurons
Algorithm
- Multiple input (on/off):
- from 1-several neurons
- Processing:
- Combination: of inputs
- Activation: on or off state
- Single output (on/off): to 1-several neurons
Artificial neurons
Algorithm
- Multiple input:
- from 1-several neurons
- Processing:
- Combination: of inputs – linear model
- Activation: activation function
- Single output: to 1-several neurons
Weighted linear combination of input:
\[ \begin{eqnarray*} z_j &=& \sum_{i} w_{i,j} a'_{i} + b_j\\ \textrm{weights}&& w_{i,j}\\ \textrm{bias}&& b_j \end{eqnarray*} \]
Activation function:
- e.g., the Sigmoid (logistic) activation function
\[a_j = \sigma(z_j)\]
The Sigmoid Neuron
Weighted linear combination of input:
- \(z_j = \sum_{i} w_{i,j} a'_{i} + b_j\)
Sigmoid/logistic activation function
- \(a_j=\sigma(z_j) = \frac{1}{1+e^{-z_j}}\)
Compare with logistic GLM
- Weighted linear combination of input:
- \(z = \sum_{i} \beta_{i} x_{i} + \alpha\)
- Sigmoid/logistic link function
- \(Pr[y=1|x] = p = \sigma(z) = \frac{1}{1+e^{-z}}\)
… or equivalently
\(\begin{eqnarray} \sigma^{-1}(p)&=&\log\left(\frac{p}{1-p}\right) =\\ logit(p) &=& \sum_{i}\beta_{i} x_{i} + \alpha \end{eqnarray}\)
Example
Let inputs be:
\(\begin{eqnarray} a'_1&=&1\\ a'_2&=&0\\ a'_3&=&1 \end{eqnarray}\)
and we have
\(\begin{eqnarray} z_1 &=& \sum_i w_{i,1}a'_i + b_1\\ a_1 &=& \sigma(z_1) \end{eqnarray}\)
\(z_1 = 0.3 \times 1 + 0.8 \times 0 + 0.2 \times 1 - 0.5 =\) \(0\)
\(a_1 = \sigma(z_1) = \frac{1}{1+e^{-0}} =\) \(0.5\)
…
So, if a sigmoid artificial neuron is just another way of doing logistic regression?
… then what’s all the fuss about?
The fuss happens when you connect several neurons into a network
Feed-forward artifical neural networks (ffANN)
Layers
“Columns” of 1-many neurons
A single Input layer
- Input neurons receives data input and passes it to next layer
1-many Hidden layer(s)
- Articial neurons process their input and deliver output to next layer
A single Output layer
- Artifical neurons process their input and deliver final output \(\hat{y}\)
- output \(\hat{y}_j = a_j\)
- Continuous \(\hat{y}\): Regression
- Discrete \(\hat{y}\): Classification
- Artifical neurons process their input and deliver final output \(\hat{y}\)
Connectivity between layers
- ffANN are fully connected (“dense layers”)
- each neuron in a layer is connected to each neurons in next layer
ffANN examples
Other drawing style, omitting \(w\) and \(b\).
Often layers are ‘boxed’
ffANN examples
layers w >1 dimension (e.g., images) – (messy!) 