Revision c75c297c03c83f9035f1800db1f9586c925314f6 (click the page title to view the current version)

Neural Networks Lecture


What is a neural network

  • The single Neuron
    • Weighted Input
    • Activation
  • The network model
    • Input/Output
    • Weights
    • Activation Function
  • The Tensor Model

Output and Loss Function

  • Classification versus Regression
    • Two-class classification (0 or 1)
    • Regression \(y=f(x)\in \mathbb{R}\)
    • Multiclass \(y=(y_1,y_2,\ldots,y_n)\)
      • \(y_i=1\) means membership in class 1
  • Soft-decision: \(y\) is a continuous variable
    • higher values are more probably one
  • Loss functiion
    • Mean-squared error (common for regression) \[L = (x-y)^2\]
    • Cross Entropy (common for classification) \[L = \log \frac{ \exp x_{y} } { \sum \exp x_i }\]
    • There are others


  • Optimisation problem
    • tune the weights to minimise the loss function
    • if the activation function is differentiable, the entire system is
    • different optimisation algorithms; trust the API or do a more advanced module

Activation Functions

  • Threshold functions
  • Approximations to the threshold function
  • Logistic: \(f(x) = \frac1{1+e^{-\beta x}}\)
  • ReLU: \(f(x)=\max(x,0)\)
    • not differentiable


Two main contenders.

  • TensorFlow
  • PyTorch
    • A replacement for NumPy to use the power of GPUs and other accelerators.
    • An automatic differentiation library that is useful to implement neural networks.

Note that PyTorch replaces NumPy; i.e. it is primarily a python tool, and operaes in the object oriented framework of python.

The reason for using PyTorch in these examples is primarily that I have lately been working off some code created by some final year students this Spring, and they happened to choose PyTorch. The choice of TensorFlow or PyTorch is otherwise arbitrary.

Sample Program



Loss Functions and Evaluation

  • Accuracy: ratio of correctly classified items
  • What is the difference between a rate and a probability?
  • Statistics
    • Standard deviation
    • Hypothesis Tests
    • Confidence Interval
  • Other heuristics

Computational Power

  • Neural Networks are Computationally Expensive
  • GPU or CPU - what’s the difference?
    • what resources do you have?
  • Remedies
    • Reduce image resolution
    • Reduce number of images
    • Reduce number of epochs
  • In particular, it is necessary to sacrifice accuracy during development and testing.
  • In the final stages you may need big datasets to achieve satisfactory results, and then you may need more computing power.