• [Up: Overview]-[Next: 3D Modelling]

Reading

• Ma 2004 Chapter 1.
• (Szeliski 2022 Chapter 1)

Session

1. Briefing Introduction
   • Goal (1) understand how we work with the subject in this module, and (2) have an overview of the challenges of the field of machine learning.
2. Exercise Python Setup (below)
   • Goal get familiar with software we are going to use throughout the semester
3. Debrief with recap on linear algebra (introductory slides continued)
   • Goal (1) tie of loose ends from the the exercise, and (2) refresh essential mathematics which should already be known.
   • If we have time we look at Change of Basis for tomorrow

Practical: Python Setup

Today’s task is to install and test a number of software packages on your computers. The goal is not to produce the output that I ask for, but to make sure the software works for you on your system.

Install Python

We will use Python 3 in this module. There is a myriad of different ways to install and use python, and you have probably used one or another in other modules already.

I make my demonstrations using command line tools, which I find simpler to use, but should feel free to use tools that work for you.

I would not recommend not to rely solely on Jupyter. Even though it has its good uses, I am not sure how easy it will be to connect USB cameras and other peripheral devices within Jupyter. You may also want to make your own modules for reusable functions, and this is easier if you work in a more general purpose environment.

Working with the command line

You need to install the following Python python and its pacakge manager pip. How you install these three packages depends on your OS. Some systems already have python and pip installed; I think this is common on MacOS.

In Debian/Ubuntu, you should use your package manager as follows.

sudo apt-get install ipython3 python3-pip

1. Note I install ipython3 here, which installs python3 implicitly. When we do this, we do not have to install ipython later with pip
2. Note also that we specify version 3 in the package names, to be certain that we do not get Python 2 which is still around.

Anaconda/Miniconda

Anaconda and Miniconda is a platform for managing python installations. This is recommended for Windows users, but I have never used it myself.

For installation, download miniconda or anaconda

• Miniconda — Conda documentation

Miniconda¶. Miniconda is a free minimal installer for conda. It is a small, bootstrap version of Anaconda that includes only conda, Python, the packages they depend on, and a small number of other useful packages, including pip, zlib and a few others.

• docs.conda.io

Create virtualenvironment with

conda create -n <name> python=<version>

e.g.

conda create -n maskinsyn python=3.8

Activate with

conda activate <name>

e.g.

conda activate maskinsyn

Install with pip install, or conda install

Creating environment

Managing environments — conda 4.10.3.post24+d808108b6 documentation

An explicit spec file is not usually cross platform, and therefore has a comment at the top such as # platform: osx-64 showing the platform where it was created. This platform is the one where this spec file is known to work.

Install Python Packages

Python packages are installed most easily using python’s own packaging tool, pip, which is independent of the OS. It is run from the command line.

Depending on how you installed pip, it may be a good idea to upgrade

pip3 install --upgrade pip

If you have not installed ipyton already, you should install it now.

pip3 install  ipython3

Note that pip installs programs under $HOME/.local/bin which you may have to add to your path.

Then we install the libraries we need. You can choose to install either in user space or as root.

User space:

pip3 install --user matplotlib numpy opencv-python

As root:

sudo pip3 install matplotlib numpy opencv-python

• numpy is a standard library for numeric computations. In particular it provides a data model for matrices with the appropriate arithmetic functions.
• matplotlib is a comprehensive library for plotting, both in 2D and 3D.
• OpenCV is a Computer Vision library, written in C++ with bindings for several different languages.

Using virtual environments

Virtual Environments, allow to manage libraries separately for each project. This is sometimes useful, but if this is the only module where you use python, it makes no difference. If you have time to spare, it is worth looking into.

Run iPython

Exactly how you run iPython may depend on you OS. In Unix-like systems we can run it straight from the command line:

ipython3

This should look something like this:

georg$ ipython3 
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: print("Hello World")                                                    
Hello World

In [2]: import numpy as np                                                      

In [3]: np.sin(np.pi)                                                           
Out[3]: 1.2246467991473532e-16

In [4]: np.sqrt(2)                                                              
Out[4]: 1.4142135623730951

In [5]:                                                                         

• What does np.sin(np.pi) and np.sqrt(2) mean? (Rewrite in mathematical symbols.)
• Is the output from np.sin(np.pi) and np.sqrt(2) as expected?

Warning.

Different versions of python and its libraries may work differently. You may therefore have to try variations to make it work. Please let me know when you have issues, whether you find a solution yourself or not.

Some 3D Operations

In this chapter, we will define a simple 3D Object and display it in python. The 3D object is an irregular tetrahedron, which has four corners and four faces.

import numpy as np
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d.art3d import Poly3DCollection

Firtsly, we define the three corners of the tetrahedron.

corners = [ [-1,0.5,0.5], [+1,0.5,0.5], [0,-0.5,0.5], [0,0.5,-0.5] ]

Each face is adjacent to three out of the four corners, and can also be defined by these corners.

face1 = [ corners[0], corners[1], corners[2] ] 
face2 = [ corners[0], corners[1], corners[3] ] 
face3 = [ corners[0], corners[2], corners[3] ] 
face4 = [ corners[1], corners[2], corners[3] ] 

To represent the 3D structure for use in 3D libraries, we juxtapose all the faces and cast it as a matrix.

vertices = np.array([face1,face2,face3,face4],dtype=float)
print(vertices)

Observe that the vertices (corners) are rows of the matrix. The mathematical textbook model has the corners as columns, and this is something we will have to deal with later.

We define the 3D object ob as follows.

ob = Poly3DCollection(vertices, linewidths=1, alpha=0.2)

The alpha parameter makes the object opaque. You may also want to play with colours:

ob.set_facecolor( [0.5, 0.5, 1] )
ob.set_edgecolor([0,0,0])

To display the object, we need to create a figure with axes.

plt.ion()
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
plt.show()

Note the plt.ion() line. You do not use this in scripts, but in ipython it means that control returns to the prompt once the figure is shown. It is necessary to continue modifying the plot after it has been created.

Note (II) depending on your platform, you may have to skip the plt.show() command and issue it at the end instead. When I use ipython I can modify the plot after it has been shown, but this is not the case in Jupyter notebook and also some other platforms, where you have to set up every detail of the plots first and then show it.

Now, we can add our object to the plot.

ax.add_collection3d(ob)

Quite likely, the object shows outside the range of the axes. We can fix this as follows:

s = [-2,-2,-2,2,2,2]
ax.auto_scale_xyz(s,s,s)

These commands make sure that the axes are scalled so that the two points (-2,-2,-2) and (2,2,2) (defined in the list s) are shown within the domain.

Rotation and Translation of 3D objects

Continuing on the previous section, our 3D object ob is defined by the vertices matrix, where all the rows are points in space. Motion is described by matrix operations on vertices.

Translation

Let us define another vector in \(\mathbb{R}^3\) and add it to each point.

translation = np.array( [ 1, 0, 0 ], dtype=float ) 
v2 = vertices + translation
print(v2)

Note that this operation does not make sense in conventional mathematics. We have just added a \(1\times3\) matrix to an \(N\times3\) matrix. How does python interpret this in terms of matrices?

To see what this means visually in 3D space, we can generate a new 3D object from v2. We use a different face colour for clarity.

ob2 = Poly3DCollection(v2, linewidths=1, alpha=0.2)
ob2.set_facecolor( [0.5, 1, 0.5] )
ob2.set_edgecolor([0,0,0])
ax.add_collection3d(ob2)

How does the new object relate to the first one in 3D space?

Rotation

In the previous test, we added a vector to the nodes in the 3D polyhedron. Let’s try instead to multiply by a matrix, like this:

theta = np.pi/6
R = np.array( [ [ np.cos(theta), -np.sin(theta), 0 ],
                [ np.sin(theta),  np.cos(theta), 0 ],
                [ 0,              0,             1 ] ], dtype=float )
v3 = np.matmul(vertices,R)
print(v3)

This gives us a new polyhedron, like this:

ob3 = Poly3DCollection(v3, linewidths=1, alpha=0.2)
ob3.set_facecolor( [1, 0.5, 0.5] )
ob3.set_edgecolor([0,0,0])
ax.add_collection3d(ob3)

Removing a 3D Object

Unfortunately, matplotlib.pyplot is not designed for interactive construction or animation of 3D graphics, so some things are little bit tricky. However, it is possible to remove an existing object from the plot. Assume we still have the objects from the last few sections.

First of all, let’s look at the objects we have plotted:

In [12]: ax.collections
Out[12]:
[<mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x7fdde8a074f0>,
 <mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x7fdde89fe400>,
 <mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x7fddf39b28b0>]

In [13]:

Here, ax is the axes system of the figure, and ax.collections is a collection of all of the objects that we have plotted. We can quite simply delete one and refresh the figure.

del ax.collections[1]
plt.show()

What happens?

Some Camera Operations

Now, we will turn from 3D objects in matplotlib to images in OpenCV. It is a good idea to restart python.

import numpy as np
import cv2 as cv
cap = cv.VideoCapture(0)
ret, frame = cap.read()

Now, ret should be True, indicating that a frame has successfully been read. If it is False, the following will not work.

gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)
cv.imshow('frame', gray)
cv.waitKey(1) 

You should see a greyscale image from your camera.

Note if you cut and paste all three lines (or the last two) in one go, it may not work. It did not when I last tested it. Presumably, imshow starts background work which has to complete before waitKey may work. Don’t be afraid to test for yourself, but if you have problems, you should try to cut and paste the imshow and the waitKey lines separately.

To close the camera and the window, we run the following.

cap.release()
cv.destroyAllWindows()

This example is digested from the tutorial on Getting Started with Videos. You may want to do the rest of the tutorial.

Testing external cameras (optional)

We have a box of external cameras for testing. It is useful to repeat the above exercise with an external camera. That means you have to change the camera selection (0) in this line:

cap = cv.VideoCapture(0)

In unix-like systems, you can use the device name, e.g. /dev/video1 in lieu of the number 0.
Thus you can double-check the camera connection with other capturing software.

Showing a co-ordinate frame (optional)

The following example may visualise rotations somewhat better than the tetrahedron given above. Try it, and observe to learn how it workse.

Step 1 Load libraries and set up a figure with axis system.

import numpy as np 
import matplotlib.pyplot as plt
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d.art3d import Poly3DCollection

ax = plt.figure().add_subplot(projection='3d')

We can visualise just the unit vectors, using a quiver plot. We make the qplot function so that we can plot a rotated version afterwards.

e1 = np.array([1,0,0])
e2 = np.array([0,1,0])
e3 = np.array([0,0,1])


def qplot(e1,e2,e3,**kw):
   ax.quiver(0, 0, 0, *e1, colors="r",**kw )
   ax.quiver(0, 0, 0, *e2, colors="g",**kw )
   ax.quiver(0, 0, 0, *e3, colors="b",**kw )

qplot(e1,e2,e3)

If you want to, you can show the plot at this stage.

Now we rotate the unit vectors, and make a new quiver plot.

R = np.array([
    [ 0.1729, -0.1468, 0.9739],
    [ 0.9739, 0.1729, -0.1468],
    [ -0.1468, 0.9739, 0.1729] ])


qplot(R@e1,R@e2,R@e3)

Note the notation for matrix multiplication with @. It is important to remember that * on matrices (numpy arrays) in python does element-wise multiplication and not matrix multiplication. Matrix multiplications can also be written as np.matmul(R,e1).

Finally, we need to scale and show the plot. The set_xlim3d and similar functions make arrow heads.

s = [-2,-2,-2,2,2,2]
ax.auto_scale_xyz(s,s,s)

ax.set_xlim3d([-2.0, 2.0])
ax.set_xlabel('X')

ax.set_ylim3d([-2.0, 2.0])
ax.set_ylabel('Y')

ax.set_zlim3d([-2, 2])
ax.set_zlabel('Z')

plt.show()

Note we did not use plt.ion() in the sequence above. What difference does that make?

If you have yet more time to spare

Do more tutorials

• Getting Started with Videos. (mentioned above)
• Basic Operations on Images
• Drawing Functions in OpenCV
• Other tutorials on OpenCV

You may also install and test the numpy-stl library. We shall use it next week.

Reflection (NOT optional)

At the end of the day, you should 1. create a directory for the module and a subdirectory for this session. 1. gather all the python code you have used to keep in .py files in the session directory. 1. Type up a short reflection note, answering + what did you learn today? + what was most interesting? + what was most challenging? + what do you most want to learn next (within the subject of the module)?

Other references

• Core Operations on images