Revision 0c624e8ca81953eb04087720851e3c22f8417c53 (click the page title to view the current version)

Neural Networks Lecture

Briefing

What is a newural network

  • The single Neuron
    • Weighted Input
    • Activation
  • The network model
    • Input/Output
    • Weights
    • Activation Function
  • The Tensor Model

Output and Loss Function

  • Classification versus Regression

MSE

\[L = (x-y)^2\]

CrossEntropy

\[L = \log \frac{ \exp x_{y} } { \sum \exp x_i }\]

Training

  • Optimisation problem
    • tune the weights to minimise the loss function
    • if the activation function is differentiable, the entire system is
    • different optimisation algorithms; trust the API or do a more advanced module

Activation Functions

  • Threshold functions
  • Approximations to the threshold function
  • Logistic: \(f(x) = \frac1{1+e^{-\beta x}}\)
  • ReLU: \(f(x)=\max(x,0)\)
    • not differentiable

Sample Problem