Heuristics

• Accuracy
• FP/FN rate
• Confusion Matrix

Exercise 1

Review the machine learning systems you studied last week. Calculate the confusion matrix for each of the systems.

• Are the errors reasonably balanced?
• Are any classes particularly difficult to detect?

Exercise 2

Reviewing again each of the systems from last week.

1. Calculate the false positive and false negative rates.
2. Estimate the standard deviation of the error rates.
3. Assess the quality of the machine learning system. Can you be confident that the error probability is satisfactory?
4. How large does the test set have to be to make a confident assessment?

Debrief

• Demo: python module/script