Revision 1ac7d9895d92c018fc96c65023512e32f72ff829 (click the page title to view the current version)

Introduction

Changes from 1ac7d9895d92c018fc96c65023512e32f72ff829 to current

---
title: Introductory Session to Machine Learing
categories: session 3D
---

+ [Up: [Overview]()]-[Next: [3D Modelling]()]

# Reading

+ Ma 2004 Chapter 1.
+ (Szeliski 2022 Chapter 1)

# Session

1.  **Briefing** Overview and History
2.  Install and Test Software
    - Simple tutorials
3.  **Debrief** questions and answers
    - recap of linear algebra
1.  **Briefing** [Introduction](http://www.hg.schaathun.net/talks/maskinsyn2023/intro/) 
    + **Goal** (1) understand how we work with the subject in this module,
      and (2) have an overview of the challenges of the field of
      machine learning.
2.  **Exercise** [Python Setup](#practical)  (below)
    + **Goal** get familiar with software we are going to use 
      throughout the semester
3.  **Debrief** with recap on linear algebra (introductory slides continued)
    + **Goal** (1) tie of loose ends from the the exercise, and (2)
      refresh essential mathematics which should already be known.
    + If we have time we look at [Change of Basis]() for tomorrow

# Practical: Python Setup {#practical}

# 1 Briefing
Today's task is to install and test a number of software packages on your
computers.  The goal is not to produce the output that I ask for,
but to make sure the software works for *you* on *your* system.

## Practical Information
## Install Python

### Information
We will use Python 3 in this module.
There is a myriad of different ways to install and use python,
and you have probably used one or another in other modules already.

+ Wiki - living document - course content
+ BlackBoard - announcements - discussion fora
+ Questions - either
    - in class
    - in discussion fora
+ Email will only be answered when there are good reasons not to use public fora.
I make my demonstrations using command line tools, which I find simpler
to use, but should feel free to use tools that work for you.

### Taught Format
I would not recommend not to rely solely on Jupyter.  Even though it has
its good uses, I am not sure how easy it will be to connect USB cameras
and other peripheral devices within Jupyter.  You may also want to make
your own modules for reusable functions, and this is easier if you work
in a more general purpose environment.

+ Sessions 4h twice a week
    - normally 1h briefing + 2h exercise + 1h debrief (may vary)
+ Exercises vary from session to session
    + mathematical exercises
    + experimental exercises
    + implementational exercises
+ **No** Compulsory Exercises
+ **Feedback in class** 
    - please ask for feedback on partial work
+ Keep a diary.  Make sure you can refer back to previous partial solution and reuse
  them.
### Working with the command line

You need to install the following [Python](https://www.python.org/downloads/)
python and its pacakge manager 
[pip](https://packaging.python.org/tutorials/installing-packages/).
How you install these three packages depends on your OS.
Some systems already have python and pip installed; 
I think this is common on MacOS.  

### Learning Outcomes

+ Knowledge
    - The candidate can explain fundamental mathematical models for digital imaging,
      3D models, and machine vision
    - The candidate are aware of the principles of digital cameras and image capture
+ Skills
    - The candidate can implemented selected techniques for object recognition and
      tracking
    - The candidate can calibrate cameras for use in machine vision systems
+ General competence
    - The candidate has a good analytic understadning of machine vision and of the 
      collaboration between machine vision and other systems in robotics
    - The candidate can exploit the connection between theory and application for 
      presenting and discussing engineering problems and solutions

### Exam

+ Oral exam $\sim 20$ min.
+ First seven minutes are *yours* 
    - make a case for your grade wrt. learning outcomes
    - your own implementations may be part of the case
    - essentially that you can explain the implementation analytically
+ The remaing 13-14 minutes is for the examiner to explore further
+ More detailed assessment criteria will be published later

## Vision

![Eye Model from *Introduction to Psychology* by University of Minnesota](Images/eye.jpg)

+ Vision is a 2D image on the retina
    + Each cell perceives the light intencity of colour of the light projected thereon
+ Easily replicated by a digital camera
    + Each pixel is light intencity sampled at a given point on the image plane

## Cognition
In Debian/Ubuntu, you should use your package manager as follows. 
```sh
sudo apt-get install ipython3 python3-pip
```

![1912 International Lawn Tennis Challenge](Images/tennis.jpg)
1. **Note** I install ipython3 here, which installs python3 implicitly.
   When we do this, we do not have to install ipython 
   [later with pip](#pip)
2. **Note also** that we specify version 3 in the package names, to be certain
   that we do not get Python 2 which is still around.

+ Human beings see 3D objects
    - not pixels of light intencity
+ We *recognise* objects - *cognitive schemata*
    - we see a *ball* - not a round patch of white
    - we remember a *tennis match* - 
      more than four people with white clothes and rackets
+ We observe objects arranged in depth
    - in front of and behind the net
    - even though they are all patterns in the same image plane
+ 3D reconstruction from 2D retina image
    - and we do not even think about how
### Anaconda/Miniconda

## Applications
Anaconda and Miniconda is a platform for managing python installations.
This is recommended for Windows users, but I have never used it myself.

- Artificial systems interact with their surroundings
    - navigate in a 3D environment
- Simpler applications
    - face recognition
    - tracking in surveillance cameras
    - medical image diagnostics (classification)
    - image retrieval (topics in a database)
    - detecting faulty products on a conveyor belt (classification)
    - aligning products on a conveyor belt 
- Other advances in AI creates new demands on vision
    - 20 years ago, walking was a major challenge for robots
    - now robots walk, and they need to see where they go ...
For installation, [download miniconda](https://docs.conda.io/en/latest/miniconda.html) or 
[anaconda](https://www.anaconda.com/products/individual)

+ [Miniconda — Conda documentation](https://docs.conda.io/en/latest/miniconda.html)

## Focus
Miniconda¶. Miniconda is a free minimal installer for conda. It is a small, bootstrap version of Anaconda that includes only conda, Python, the packages they depend on, and a small number of other useful packages, including pip, zlib and a few others.

- Artificial systems interact with their surroundings
    - navigate in a 3D environment
- This means
    - Geometry of multiple views
    - Relationship between theory and practice
    - ... between analysis and implementation
- Mathematical approach
    - inverse problem; 3D to 2D is easy, the inverse is hard
    - we need to understand the geometry to know what we program
+ docs.conda.io

##  History 
Create virtualenvironment with

- 1435: *Della Pictura* - first general treatise on perspective
- 1648 Girard Desargues - projective geometry
- 1913 Kruppa: two views of five points suffice to find
    - relative transformation 
    - 3D location of the points 
    - (up to a finite number of solutions)
- mid 1970s: first algorithms for 3D reconstruction
- 1981 Longuet-Higgins: linear algorithm for structure and motion
- late 1970s E. D. Dickmans starts work on vision-based autonomous cars
    - 1984 small truck at 90 km/h on empty roads
    - 1994: 180 km/h, passing slower cars
```sh
conda create -n <name> python=<version>
```

## Python
e.g.

- Demos and tutorials in Python
    - you can use whatever language you want
    - we avoid Jupyter to make sure we can use camera and interactive displays easily
- Demos and help on Unix-like system (may or may not include Mac OS)
- In the exercise sessions
    - install necessary software
    - use the tutorials to see that things work as expected
- In the debrief, we will start briefly on the mathematical modelling
```sh
conda create -n maskinsyn python=3.8
```

# 2 Lab Practice
Activate with

The most important task today is to install and test a number
of software packages needed.  
```sh
conda activate <name>
```

## Install Python
e.g.

We will use Python 3 in this module.
You need to install the following.
```sh
conda activate maskinsyn
```

1. [Python](https://www.python.org/downloads/)
2. [pip](https://packaging.python.org/tutorials/installing-packages/).
    (the package manager for python)
3. [iPython](https://ipython.org/install.html)
    (a more convenient interactive interpreter)
Install with `pip install`, or `conda install`

How you install these three packages depends on your OS.
In most distroes you can use the package system to install all of this,
for instance in Debian/Ubuntu: 
[Creating environment](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands)

```sh
sudo apt-get install ipython3 python3-pip
```
[Managing environments — conda 4.10.3.post24+d808108b6 documentation](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands)

(Python is installed automatically as a dependency of iPython3.
Note, you have to specify version 3 in the package names,
lest you get Python 2.)
An explicit spec file is not usually cross platform, and therefore has a comment at the top such as # platform: osx-64 showing the platform where it was created. This platform is the one where this spec file is known to work.

## Install Python Packages
### Install Python Packages {#pip}

Python packages are installed most easily using python's own
packaging tool, pip, which is independent of the OS.
It is run from the command line.

Depending on how you installed pip, it may be a good idea to upgrade

```sh
pip3 install --upgrade pip
```

If you have not installed ipyton already, you should install it now.

```sh
pip3 install  ipython3
```

Note that `pip` installs programs under `$HOME/.local/bin` which you may have to
add to your path.

Then we install the libraries we need.
You can choose to install either in user space or as root.

User space:

```sh
pip3 install --user matplotlib numpy opencv-python
```

As root:

```sh
sudo pip3 install matplotlib numpy opencv-python
```

+ numpy is a standard library for numeric computations.
  In particular it provides a data model for matrices with
  the appropriate arithmetic functions.
+ matplotlib is a comprehensive library for plotting, both in 2D and 3D. 
+ [OpenCV](https://opencv.org/) 
  is a Computer Vision library, written in C++ with bindings for
  several different languages.

A third installation alternative is to use
### Using virtual environments

[Virtual Environments](https://docs.python.org/3/tutorial/venv.html),
which allows you to manage python versions and dependencies separately
for each project.  This may be a good idea if you have many python 
projects, but if this is your first one, it is not worth the hassle.
allow to manage libraries separately for each project.
This is sometimes useful, but if this is the only module where you use
python, it makes no difference.
If you have time to spare, it is worth looking into.

## Run iPython

Exactly how you run iPython may depend on you OS.
In Unix-like systems we can run it straight from the command line:

```sh
ipython3
```

This should look something like this:

```
georg$ ipython3 
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: print("Hello World")                                                    
Hello World

In [2]: import numpy as np                                                      

In [3]: np.sin(np.pi)                                                           
Out[3]: 1.2246467991473532e-16

In [4]: np.sqrt(2)                                                              
Out[4]: 1.4142135623730951

In [5]:                                                                         
```

+ What does `np.sin(np.pi)` and `np.sqrt(2)` mean?  (Rewrite in mathematical symbols.)
+ Is the output from `np.sin(np.pi)` and `np.sqrt(2)` as expected?

## Warning.

Different versions of python and its libraries may work
differently.  You may therefore have to try variations to 
make it work.  Please let me know when you have issues,
whether you find a solution yourself or not.

## Some 3D Operations

In this chapter, we will define a simple 3D Object and display it in python.
The 3D object is an irregular tetrahedron, which has four corners and four
faces.


```python
import numpy as np
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d.art3d import Poly3DCollection
```

Firtsly, we define the three corners of the tetrahedron.

```python
corners = [ [-1,0.5,0.5], [+1,0.5,0.5], [0,-0.5,0.5], [0,0.5,-0.5] ]
```

Each face is adjacent to three out of the four corners, and can
also be defined by these corners.

```python
face1 = [ corners[0], corners[1], corners[2] ] 
face2 = [ corners[0], corners[1], corners[3] ] 
face3 = [ corners[0], corners[2], corners[3] ] 
face4 = [ corners[1], corners[2], corners[3] ] 
```

To represent the 3D structure for use in 3D libraries,
we juxtapose all the faces and cast it as a matrix.


```python
vertices = return np.array(face1+face2+face3+face4,dtype=float)
vertices = np.array([face1,face2,face3,face4],dtype=float)
print(vertices)
```

Observe that the vertices (corners) are rows of the matrix.
The mathematical textbook model has the corners as columns, and
this is something we will have to deal with later.


We define the 3D object `ob` as follows.

```python
ob = Poly3DCollection(vertices, linewidths=1, alpha=0.2)
```

The `alpha` parameter makes the object opaque.
You may also want to play with colours:

```python
ob.set_facecolor( [0.5, 0.5, 1] )
ob.set_edgecolor([0,0,0])
```

To display the object, we need to create a figure with axes.

```python
plt.ion()
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
plt.show()
```

Note the `plt.ion()` line.  You do not use this in scripts, but in
**Note** the `plt.ion()` line.  You do not use this in scripts, but in
`ipython` it means that control returns to the prompt once the 
figure is shown.  It is necessary to continue modifying the plot after
it has been created.

**Note (II)**
depending on your platform, you may have to skip the `plt.show()`
command and issue it at the end instead.
When I use `ipython` I can modify the plot after it has been shown,
but this is not the case in Jupyter notebook and also some other platforms,
where you have to set up every detail of the plots first and then show it.

Now, we can add our object to the plot.

```python
ax.add_collection3d(ob)
```

Quite likely, the object shows outside the range of the axes.
We can fix this as follows:

```
```python
s = [-2,-2,-2,2,2,2]
ax.auto_scale_xyz(s,s,s)
```

These commands make sure that the axes are scalled so that the two
points `(-2,-2,-2)` and `(2,2,2)` (defined in the list `s`)
are shown within the domain.

## Rotation and Translation of 3D objects

**TODO**
Continuing on the previous section, our 3D object `ob` is defined 
by the `vertices` matrix, where all the rows are points in space.
Motion is described by matrix operations on `vertices`.

### Translation

Let us define another vector in $\mathbb{R}^3$ and add it
to each point.

```python
translation = np.array( [ 1, 0, 0 ], dtype=float ) 
v2 = vertices + translation
print(v2)
```

Note that this operation does not make sense in conventional
mathematics.  We have just added a $1\times3$ matrix to an $N\times3$ matrix.
How does python interpret this in terms of matrices?

To see what this means visually in 3D space, we can generate
a new 3D object from `v2`.  We use a different face colour 
for clarity.

```python
ob2 = Poly3DCollection(v2, linewidths=1, alpha=0.2)
ob2.set_facecolor( [0.5, 1, 0.5] )
ob2.set_edgecolor([0,0,0])
ax.add_collection3d(ob2)
```

How does the new object relate to the first one in 3D space?

### Rotation

In the previous test, we added a vector to the nodes in the
3D polyhedron.  Let's try instead to multiply by a matrix, like this:

```python
theta = np.pi/6
R = np.array( [ [ np.cos(theta), -np.sin(theta), 0 ],
                [ np.sin(theta),  np.cos(theta), 0 ],
                [ 0,              0,             1 ] ], dtype=float )
v3 = np.matmul(vertices,R)
print(v3)
```

This gives us a new polyhedron, like this:

```python
ob3 = Poly3DCollection(v3, linewidths=1, alpha=0.2)
ob3.set_facecolor( [1, 0.5, 0.5] )
ob3.set_edgecolor([0,0,0])
ax.add_collection3d(ob3)
```

### Removing a 3D Object

Unfortunately, `matplotlib.pyplot` is not designed for interactive
construction or animation of 3D graphics, so some things are little
bit tricky.  However, it is possible to remove an existing object
from the plot.  Assume we still have the objects from the last few
sections.

First of all, let's look at the objects we have plotted:

```
In [12]: ax.collections
Out[12]:
[<mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x7fdde8a074f0>,
 <mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x7fdde89fe400>,
 <mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x7fddf39b28b0>]

In [13]:
```

Here, `ax` is the axes system of the figure, and `ax.collections`
is a collection of all of the objects that we have plotted.
We can quite simply delete one and refresh the figure.

```python
del ax.collections[1]
plt.show()
```

What happens?

## Some Camera Operations

Now, we will turn from 3D objects in matplotlib to images
in OpenCV.  It is a good idea to restart python.

```python
import numpy as np
import cv2 as cv
cap = cv.VideoCapture(0)
ret, frame = cap.read()
```

Now, `ret` should be `True`, indicating that a frame has
successfully been read.  If it is `False`, the following
will not work.

```python
gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)
cv.imshow('frame', gray)
cv.waitKey(1) 
```

You should see a greyscale image from your camera.

**Note** if you cut and paste all three lines (or the last two)
in one go, it may not work.
It did not when I last tested it.
Presumably, `imshow` starts background work which has to complete
before `waitKey` may work.
Don't be afraid to test for yourself, but if you have problems, you should
try to cut and paste the `imshow` and the `waitKey` lines separately.

To close the camera and the window, we run the following.

```python
cap.release()
cv.destroyAllWindows()
```

# 3 Debrief
This example is digested from the tutorial on
[Getting Started with Videos](https://docs.opencv.org/master/dd/d43/tutorial_py_video_display.html).
You may want to do the rest of the tutorial.

## Vectors and Points
### Testing external cameras (optional)

+ A *point* in space $\mathbf{X} = [X_1,X_2,X_3]^\mathrm{T}\in\mathbb{R}^3$
+ A *bound vector*, from $\mathbf{X}$ to $\mathbf{Y}$: $\vect{\mathbf{XY}}$
+ A *free vector* is the same difference, but without any specific anchor point
   + represented as $\mathbf{Y} - \mathbf{X}$ 
+ Set of free vectors form a linear vector space
   + **note** points do not
   + The sum of two vectors is another vector
   + The sum of two points is not a point
We have a box of external cameras for testing.
It is useful to repeat the above exercise with an external camera.
That means you have to change the camera selection (0) in this line:
```
cap = cv.VideoCapture(0)
```
In unix-like systems, you can use the device name, e.g.
`/dev/video1` in lieu of the number 0.  
Thus you can double-check the camera connection with other
capturing software.

## Dot product (inner product)
## Showing a co-ordinate frame (optional)

$$x=\begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix}\quad
  y=\begin{bmatriy}y_1\\y_2\\y_3\end{bmatrix}$$
The following example may visualise rotations somewhat better
than the tetrahedron given above.  Try it, and observe to learn
how it workse.

**Inner product**
$$\langle x,y\rangle = x^\mathrm{T}y = x_1y_1+x_2y_2+x_3y_3$$
**Step 1** Load libraries and set up a figure with axis system.

Euclidean **Norm**
$$||x|| = \sqrt{\langle x,x\rangle}$$

**Orthogonal vectors** when $\langle x,y\rangle=0$
```python
import numpy as np 
import matplotlib.pyplot as plt
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d.art3d import Poly3DCollection

## Cross product
ax = plt.figure().add_subplot(projection='3d')
```

$$x\times y = 
\begin{bmatrix}
  x_2y_3 - x_3y_2 \\
  x_3y_1 - x_1y_3 \\
  x_1y_2 - x_2y_1 
\end{bmatrix} \in \mathbb{R}^3$$
We can visualise just the unit vectors, using a
quiver plot.  We make the `qplot` function so
that we can plot a rotated version afterwards.

Observe that
```python
e1 = np.array([1,0,0])
e2 = np.array([0,1,0])
e3 = np.array([0,0,1])

+ $y\times x = -x\times y$
+ $\langle x\times y, y\rangle= \langle x\times y, x\rangle

$$x\times y = \hat xy \quad\text{where}\quad \hat x =  
\begin{bmatrix}
  0 -x_3 x_2 \\
  x_3 0 -x_1 \\
  -x_2 x_1  0
\end{bmatrix} \in \mathbb{R}^{3\times3}$$
def qplot(e1,e2,e3,**kw):
   ax.quiver(0, 0, 0, *e1, colors="r",**kw )
   ax.quiver(0, 0, 0, *e2, colors="g",**kw )
   ax.quiver(0, 0, 0, *e3, colors="b",**kw )

$\hat x$ is a **skew-symmetric** matrix because $\hat x=-\hat x^\mathrm{T}$
qplot(e1,e2,e3)
```

## Right Hand Rule
If you want to, you can show the plot at this stage.

**TODO**
Now we rotate the unit vectors, and make a new quiver
plot.

## Skew-Symmetric Matrix
```python
R = np.array([
    [ 0.1729, -0.1468, 0.9739],
    [ 0.9739, 0.1729, -0.1468],
    [ -0.1468, 0.9739, 0.1729] ])

**TODO**

## Change of Basis
qplot(R@e1,R@e2,R@e3)
```

**TODO**
**Note** the notation for matrix multiplication with `@`.
It is important to remember that `*` on matrices 
(numpy arrays) in python does *element-wise* multiplication
and not matrix multiplication.
Matrix multiplications can also be written as
`np.matmul(R,e1)`.

Finally, we need to scale and show the plot.
The `set_xlim3d` and similar functions make
arrow heads.

```python
s = [-2,-2,-2,2,2,2]
ax.auto_scale_xyz(s,s,s)

ax.set_xlim3d([-2.0, 2.0])
ax.set_xlabel('X')

ax.set_ylim3d([-2.0, 2.0])
ax.set_ylabel('Y')

ax.set_zlim3d([-2, 2])
ax.set_zlabel('Z')

plt.show()
```

**Note** we did not use `plt.ion()` in the sequence above.
What difference does that make?

### If you have yet more time to spare

Do more tutorials

+ [Getting Started with Videos](https://docs.opencv.org/master/dd/d43/tutorial_py_video_display.html).
  (mentioned above)
+ [Basic Operations on Images](https://docs.opencv.org/4.x/d3/df2/tutorial_py_basic_ops.html)
+ [Drawing Functions in OpenCV](https://docs.opencv.org/4.x/dc/da5/tutorial_py_drawing_functions.html)
+ [Other tutorials on OpenCV](https://docs.opencv.org/4.x/d6/d00/tutorial_py_root.html)

You may also install and test the numpy-stl library.
We shall use it [next week](3D Objects in Python#stl-files-and-the-stl-library).

## Reflection (NOT optional) {#reflection}

At the end of the day, you should
1. create a directory for the module and a subdirectory for this session.
1. gather all the python code you have used to keep in `.py` files in
   the session directory.
1. Type up a short reflection note, answering
    + what did you learn today?
    + what was most interesting?
    + what was most challenging?
    + what do you most want to learn next (within the subject of the module)?


## Other references

- [Core Operations](https://docs.opencv.org/4.5.1/d7/d16/tutorial_py_table_of_contents_core.html)
	  on images