Revision 444e5a3daf141740f8beedb68560a6eaf9c0fd3f (click the page title to view the current version)

Neural Networks Lecture


What is a newural network

  • The single Neuron
    • Weighted Input
    • Activation
  • The network model
    • Input/Output
    • Weights
    • Activation Function
  • The Tensor Model

Output and Loss Function

  • Classification versus Regression


\[L = (x-y)^2\]


\[L = \log \frac{ \exp x_{y} } { \sum \exp x_i }\]


  • Optimisation problem
    • tune the weights to minimise the loss function
    • if the activation function is differentiable, the entire system is
    • different optimisation algorithms; trust the API or do a more advanced module

Activation Functions

  • Threshold functions
  • Approximations to the threshold function
  • Logistic: \(f(x) = \frac1{1+e^{-\beta x}}\)
  • ReLU: \(f(x)=\max(x,0)\)
    • not differentiable


Two main contenders.

  • TensorFlow
  • PyTorch
    • A replacement for NumPy to use the power of GPUs and other accelerators.
    • An automatic differentiation library that is useful to implement neural networks.

Note that PyTorch replaces NumPy; i.e. it is primarily a python tool, and operaes in the object oriented framework of python.

The reason for using PyTorch in these examples is primarily that I have lately been working off some code created by some final year students this Spring, and they happened to choose PyTorch. The choice of TensorFlow or PyTorch is otherwise arbitrary.

Sample Program