# Functional Programming and Intelligent Algorithms

## Tutorial 1.4: The Single Neuron

(Sist oppdatert: 1 January 2015)

### Overview

• Reading: Stephen Marsland: Chapter 1-3

In this tutorial we shall implement our first learning algorithm, namely a single neuron. The celebrate artificialneural networks (ANN) are built up of numerous neurons, so this tutorial is the first step.

### Problem 1: Single Neuron and Perceptron Training

Marsland discusses the perceptron as a collection of individual neurons. Observe that the neurons are completely independent. Each one receives the same input vector, but produces its own scalar output. Combining several neurons into one perceptron gives us vector output.

In this Problem we will implement only a single neuron with the perceptron training algorithm. Building a multi-neuron perceptron is a relatively simple extension which we consider later.

Marsland Chapter 3 gives all the details you need to understand the neuron and the perceptron.

#### Step 1: Input Vector

At the conceptual (mathematical) level, the neuron receives a real vector as input. The output is 0.0 or 1.0, which we also consider as a real number.

What data type should be used in Haskell to represent the input and the output from the neuron?

You may define your own data types or type aliases for input and output if you want to, but it is not necessary.

#### Step 2: Data Type

Define a data type `Neuron` to represent a single neuron, recording all the weights.

Think through the following questions.

• What information must be stored as part of the neuron?
• Is it possible to use a list? Why/why not?
• Is it possible to use a tuple? Why/why not?
• What data type should be used for the weights? Why?

#### Step 3: Initialisation

We need a function to create a new, pristine neuron. In a production system, this should be done randomly, but randomness is non-trivial, so we have to return to that later. For the time being, we are happy to initialise the neuron with small constant weights (say 0.01).

Define a function `initNeuron :: Integer -> Neuron` which returns a neuron with small non-zero weights (say 0.01). The integer input specifies the dimension of the input vector to be used with the neuron.

Test the function in `ghci`. Does the function give you what you expect?

#### Step 4: Recall

In operation, the so-called recall, the neuron receives an input vector and produces a scalar output. We need the following function to do this,

``` recall :: Neuron -> InputVector -> OutputValue ```

The algorithm (formula) is defined by Marsland, p. 47. Remember to include the thresholding, so that the output is 0 or 1, and remember that we have one more weight than inputs, and this extra weight is multiplied by -1.

Define the `recall` function. Replace InputVector and OutputVector with the types you decided to use in Step 1.

We need to test the function. Start `ghci` and try the following ``` recall (initNeuron 3) [ 1.0, 0.5, -1.0 ] ``` If you have used another datatype for the input vector, other than list, you need to change the test call appropriately. Do you get the expected output?

Obviously, you do not learn all that you want to know from the above test, but at least you get to check for type errors. Develop your own test, by manually defining a test neuron with other weights, and use that in lieu of `initNeuron`.

#### Step 5: Training

The training function receives an old neuron, together with an input vector and a corresponding output value, and it returns a new neuron, with updated weights. Additionally, it takes the training factor η. Thus we need a function with the following signature.

``` trainOne :: Double -> InputVector -> OutputValue -> Neuron -> Neuron ```

This function considers only one input/output pair and updates the neuron. The complete `train` function, which we consider later, will use `trainOne` to consider multiple input/output pairs and train the neuron iteratively.

Define the `trainOne` function. Replace InputVector and OutputVector with the types you decided to use in Step 1.

We need to test the function similarly to what we did in the previous step. Start `ghci` and try the following ``` trainOne 0.5 [ 1.0, 0.5, -1.0 ] 1.0 (initNeuron 3) ``` Do you get the expected output?

#### Step 6: Training on a Set of Vectors

Using the above `trainOne` function, we want to design a `trainSet` function which takes a list of input vectors instead of a the single one, and trains the neuron on each input vector in turn. This gives this signature:

``` trainSet :: Double -> [InputVector] -> OutputValue -> Neuron -> Neuron ```

Define the `trainSet` function. Replace InputVector and OutputVector with the types you decided to use in Step 1.

Test the function as you did with `trainOne`, but replace the input vector with a list of two input vectors (of you choice), each of length 3.

#### Step 7: Complete Training Function

It is usually not sufficient to run the training once only. Usually, we want to repeat the `trainSet` operation T times. In other words, we want a function with the following signature:

``` train :: Integer -> Double -> [InputVector] -> OutputValue -> Neuron -> Neuron ```

The `train` function repeatedly applies `trainSet` T times. Each round uses the output neuron from the previous application as input.

Define the `trainSet` function. Replace InputVector and OutputVector with the types you decided to use in Step 1.

Test the function using the same test data as you used for `trainSet`. Try both T=2 and T=5. replace the input vector with a list of two input vectors (of you choice), each of length 3.

#### Step 8: Testing

A simple test to device is to take a simple function and try] to predict whether it is positive or negative. Take, e.g. $\mathrm{f\left(x,y,z\right)}={\mathrm{\left(x-1.0\right)}}^{4}+{\mathrm{\left(y-1.0\right)}}^{5}-z-\mathrm{1.0}$ Now you can choose a couple of $\mathrm{\left(x,y,z\right)}$ points and calculate the corresponding function value. This gives you input and output values to train your perceptron.

You may also want to test your perceptron on data which was not used for training. Choose a few other $\mathrm{\left(x,y,z\right)}$ points, and you can easily calculate both $\mathrm{f\left(x,y,z\right)}$ and `recall [x,y,z] n` where `n` is a trained neuron.

### Problem 2: Multi-Neuron Perceptrons

It is not necessary to solve this problem at this stage, but we will need it eventually. Please postpone it if you are behind schedule.

#### Step 1: Data Type

Define a data type `Layer` to represent a multi-neuron perceptron.

What data type can be used to hold a set of neurons?

The name `layer' will make sense when we advance to more complex neural networks. The perceptron consists of a single layer only, but other neural networks will be lists of layers.

#### Step 2: Initialisation

Define a function `initLayer` to return a perceptron (layer) where all weights in all neurons is set to some small, constant, non-zero value. Remember arguments so that the user can choose both the number of neurons in the layer and the number of inputs.

#### Step 3: Recall

Define a function `recallLayer` which does a recall for each neuron in the layer, and returns a list of output values.

#### Step 4: Training

Generalise each of the training functions `trainOne`, `trainSet`, and `train` for perceptrons. The training functions for perceptrons have to apply the corresponding training function for each neuron in the layer.

### Epilogue

You have just implemented your first classifier. Well done.

However, this prototype leaves much to be desired.

1. We cannot initialise with random weights.
2. We have to type in the data for training and for testing.

As you can see, we have to go back and learn some more techniques in Haskell. First of all, we will learn I/O in the next tutorial, to be able to read complete data sets from file, both for training and for testing.