Task 6 - EN (Original)

Task 6

Let us consider a multi-label classification problem. For each class \(c\), \(TP_c\) represents the instances that are in class \(c\) correctly identified as belonging to \(c\); \(TN_c\) represents the instances that are not in class \(c\) correctly identified as not belonging to \(c\); \(FP_c\) represents the instances that are in not class \(c\) incorrectly labeled as belonging to \(c\); \(FN_c\) represents the instances that are in class \(c\) incorrectly labeled as not belonging to \(c\).

We have the following metrics:

• per-class recall:
  \[
  \mathrm{Recall}_c = \frac{TP_c}{TP_c+FN_c}=P(\mathrm{pred} = c\mid\mathrm{true}=c);
  \]
• per-class precision:
  \[
  \mathrm{Precision}_c = \frac{TP_c}{TP_c+FP_c}=P(\mathrm{true}=c\mid\mathrm{pred} =c);
  \]
• per-class specificity:
  \[
  \mathrm{Spec}_c = \frac{TN_c}{TN_c+FP_c}=P(\mathrm{pred}\neq c\mid\mathrm{true}\neq c);
  \]
• for each class-dependent error metric \(M_c\), the macro average error metric is the arithmetic mean of the class specific scores
  \[
  \mathrm{Macro}-M=\frac{\sum_c M_c}{\textrm{number of classes}};
  \]
• for a distribution \(\pi=(\pi_c)\) of the labels, the accuracy:
  \[
  \mathrm{acc}(\pi)=\sum_c\pi_c\mathrm{Recall}_c;
  \]
• if \(C = (C_{ij})\) is a cost matrix where \(C_{ij} = \mathrm{Cost}(\mathrm{pred} = j\mid\mathrm{true} = i)\), the expected cost is
  \[
  \mathbb{E}_\pi(C) = \sum_{i,j}\pi_i P(\mathrm{pred} = j\mid\mathrm{true} = i)C_{ij}.
  \]
  You can assume that every class has nonzero support and the classifier makes both correct and incorrect predictions for every class (non-degeneracy).

1. Let us assume that you are in a binary classifier, with classes \(0\) and \(1\). Prove that, if
   \[
   \mathrm{accuracy}=\mathrm{precision}_1=\mathrm{recall}_1,
   \]
   the support of the two classes must be the same.

[5 marks]

For the remaining parts, you can assume the following context.

Context

A public-health triage system classifies incoming messages into one of three classes:
\[
U = \text{Urgent}, \quad
S = \text{Semi-urgent}, \quad
N = \text{Non-urgent}.
\]

The training dataset was intentionally enriched with urgent and semi-urgent cases.  
On the training set, the classifier produced the following confusion matrix
(rows correspond to the true label, columns to the predicted label):

The corresponding training label distribution is
\[
\pi_{\text{train}} =
\big(P(U)=0.25,\; P(S)=0.35,\; P(N)=0.40\big).
\]

At deployment time, the real-world label distribution is different:
\[
\pi_{\text{dep}} =
\big(P(U)=0.05,\; P(S)=0.15,\; P(N)=0.80\big).
\]

You may assume that the classifier's conditional behavior is stable:
\[
P_{\text{train}}(\mathrm{pred} = j \mid \mathrm{true} = i) = P_{\text{dep}}(\mathrm{pred} = j \mid \mathrm{true} = i)
\quad \text{for all } i,j \in \{U,S,N\}.
\]

We are given the following cost matrix
(cost of predicting the column label when the true label is the row label):

\[
C(\mathrm{pred} = j \mid \mathrm{true} = i) =
\begin{array}{c|ccc}
 & U & S & N \\ \hline
U & 0 & 20 & 100 \\
S & 5 & 0 & 10 \\
N & 1 & 1 & 0
\end{array}
\]

2. Compute \(\mathrm{Precision}_U\) under both \(\pi_{\text{train}}\) and \(\pi_{\text{dep}}\).
   Explain why precision changes under distribution shift while recall does not.

[5 marks]

3. Consider two strategies for choosing the deployed decision rule:
   1. maximizing training accuracy;
   2. minimizing expected deployment cost.

Explain why strategy (1) can lead to a higher expected cost at deployment than strategy (2).

[3 marks]

[Total 13 marks]