Revision 66aa985a4fdd9639cbc3dc226df51d8b765cbece (click the page title to view the current version)
Evaluation and Statistics
Heuristics
- Accuracy
- FP/FN rate
- Confusion Matrix
Exercise 1
Review the machine learning systems you studied last week. Calculate the confusion matrix for each of the systems.
- Are the errors reasonably balanced?
- Are any classes particularly difficult to detect?
Exercise 2
Reviewing again each of the systems from last week.
- Calculate the false positive and false negative rates.
- Estimate the standard deviation of the error rates.
- Assess the quality of the machine learning system. Can you be confident that the error probability is satisfactory?
- How large does the test set have to be to make a confident assessment?
Debrief
- Demo: python module/script