Confusion matrix

Different day, same bullshit! OR different expression as potato patatas! If the devil is in the details, then you’re here to meet the devil.

Justice systems in Western countries would rather a guilty person walks free than imprison an innocent person. are designed to make Type II errors. They fail to reject the null hypothesis when it is false. Even if such justice systems were free of human corruption, they will still produce injustice by causing so many innocent people undue pain because of their fundamental design. Hence, the common principle and expression, “innocent until proven guilty” - even if in reality, the person in question is guilty.

Unless we all attain omniscient knowledge, there is no way to avoid this! Human knowledge is limited.

My interest is not really justice systems, even though these fundamental principles apply. Rather, my interests hover around:

1. How easy it is to either forget or mistakenly specify an error as Type I when it should be Type II and vice versa. Esu mischievously puzzles us!
2. Despite (1) above, how this basic idea of Type I and II errors has wide reaching applications; for instance, probability, data science, machine learning (ML), and artificial intelligence (AI).
3. Other related topics as one’s mind may bring up from time to time.

Added to the elusive nature of this idea, is the unnecessary complexity that comes with different domains calling the same thing different names. As we already know, statisticians refer to the idea as Type I and Type II. Open many books on data science and machine learning, and you’re likely to come across the term, confusion matrix. Is this different from or even related to complexity curve? Confusing indeed! What about such the ROC curve? Where does it fit in? Is it the same thing as complexity curve? Is it related to confusion matrix? Are these terms just some shitty system to keep uninitiated out in order to protect the initiated?

To my understanding, there is no difference between the core idea of Type I/II errors and what is contained in a confusion matrix. The table below is a one-on-one mapping of both concepts.

A confusion matrix is a \(2 \times 2\) table mostly used to evaluate the performance of classification algorithms in machine learning (ML). A popular example of such ML algorithms is \(k\)-NN. Note however, that confusion matrices can be extended to \(N \times N\) dimensions or multi-class classifications. Hence, they aren’t limited to binary classification problems.

	Decision: Do not reject H₀ (Predicted: NO)	Decision: Reject H₀ (Predicted: YES)
Actual: H₀ is True (Actual: NO)	Correct Decision (True Negative)	Type I Error (False Positive)
Actual: H₀ is False (Actual: YES)	Type II Error (False Negative)	Correct Decision (True Positive)

Interpreting the confusion matrix

Think of this section as Type I/II error recap.

To interprete the confusion matrix above, one can think of “Actual NO” as the true state of the world. That is, in actuality, the person did not commit the crime. “Predicted No” refers to the conclusion of the judge or what a ML model predicted, given the data (sample or evidence) it was exposed to. If the conclusion is “NO”, then a correct prediction was made by either the judge or the ML model. However, if the conclusion was “YES”, then a wrong decision or prediction was made by either the judge or the ML model (Type I error: false positive).

What if the true state of the world were “Actual YES” but the ML model predicted “no”? Well, Type II error would have occurred (flase negative). In reality, does this happen and does it happen frequently? Yes and yes! My idea is that this cannot be avoided and analysts simply fix type I/II error rates.

–insert text on the applied statisticians approach–

Let’s say a government agency is to reduce processing time on small business grant applications. Its has built an ML model that classifies grant applications as either fraudulent or non-fraudulent. Applications classified as fraudulent grant applications receive an automatic rejection, thereby reducing processing time. Applications the ML model classified as non-fraudulent applications are further processed by humans (government officers). This agency may prefer a ML model that is more likely to commit false negative (Type II) errors compared to if it were an agency classifying cancer patients. The life consequences of an ML model classifying a person who actually has cancer as not having cancer is more dire compared to business grant applications.

Recognising our human limitations, applied statisticians (styled as data analysts, data scientists, ML engineers, etc) split datasets into training, testing - and –insert— datasets. The core idea is to measure the an algorithm’s accuracy with training data, before exposing the same algorithm to new (testing) datasets. This is not and can never be foolproof as . All too often, predictive ML models perform well on testing datasets only to fall short when exposed to new data. Such ML models are referred to as overfitted models. They explain a lot of the variation (high variance) in the training data but do not perform likewise on new data. The opposite can also occur: underfitted ML models will explain very little of the variation in training data and the testing data (highly biased). How does bias-variance tradeoff connect to the bias-variance tradeoff? Since overfitted models capture noise in the training dataset, such models most likely commit Type I/II errors if exposed to new data.

Why do I say, either? Because these error rates cannot be reduced to zero in parallel. In practice, it boils down to choosing your poison and optimising for it. If in an analytical context, there is preference (think tradeoff ) for making Type II error (false negative) over Type I error (false positive), analysts can optimise for it. The analysts cannot expect that within that context, they can reduce Type I/II errors to close to zero.

To reduce the bias-variance trade-off, applied statisticans typically adopt an optimisation technique called cross-validation.

An example: Confusion matrix for a kNN model

Bias-variance trade off

Why use the word, ““confusion” in confusion matrix?”

—here trace the origin or etymology of “confusion matrix”—

Compare it to the Type I and Type II error table in my previous post.

	\(H_0\) is true.	\(H_0\) is false.
Do not reject \(H_0\) (\(p \geq \alpha\))	We made the correct decision to not reject \(H_0\). We concluded there is no sign of an effect.	Type II error rate (\(p = \beta\)). Do not reject \(H_0\) when it is false. False negative: We missed the effect.
Reject \(H_0\) (\(p < \alpha\))	Type I error rate (\(p = \alpha\)). Reject \(H_0\) when it is true. False positive: We wrongly claimed an effect.	Correctly reject \(H_0\). We found an effect.

In essence, a confusion matrix is a Type I, Type II error table applied to machine learning (ML) models. Why? Because humans, such as courtroom judges, aren’t the only ones susceptible to making Type I, II errors. ML models are just as susceptible! So, hopefully, that settles that. What