(2.5 pts) 10 is the value of \(w\) that minimizes...
Chapter 1: Introduction to Supervised Learning
Topics: squared loss and the constant model, absolute loss, comparing loss functions
Problems below are collected from past exams; each links back to its full exam. Solutions are in the dropdowns.
Problems
- FA25 MT1 · Problem 1
- FA25 MT1 · Problem 2
- FA25 Final · Problem 1
- WN26 MT1 · Problem 1
- WN26 MT1 · Problem 7
- WN26 Final · Problem 1
- SP26 MT1 · Problem 1
- SP26 MT1 · Problem 2
- SP26 Final · Problem 1
FA25 MT1 · Problem 1: Consider the Following... 15 pts
From FA25 MT1
Consider the following dataset of \(n = 9\) values.
| \(y_1\) | \(y_2\) | \(y_3\) | \(y_4\) | \(y_5\) | \(y_6\) | \(y_7\) | \(y_8\) | \(y_9\) |
|---|---|---|---|---|---|---|---|---|
| \(7\) | \(8\) | \(10\) | \(10\) | \(11\) | \(13\) | \(14\) | \(17\) | \(27\) |
Suppose we’d like to find the optimal parameter, \(w^{\ast}\), for the constant model \(h(x_i) = w\), given this dataset of 9 values.
In parts a) through f), choose the empirical risk function \(R(w)\) that the given value of \(w^{\ast}\) is the minimizer of, for this particular dataset. If you believe the given value of \(w^{\ast}\) does not minimize any of the five options, select N/A.
Option 1: \(\displaystyle R(w) = \frac{1}{n} \sum_{i = 1}^n (y_i - w)^2\)
Option 2: \(\displaystyle R(w) = \frac{1}{n} \sum_{i = 1}^n (27y_i - 13w)^2\)
Option 3: \(\displaystyle R(w) = \frac{1}{n} \sum_{i = 1}^n 13|y_i - w|\)
Option 4: \(\displaystyle R(w) = \frac{1}{n} \sum_{i = 1}^n \begin{cases} 13 & \text{if } y_i = w \\ 27 & \text{if } y_i \neq w \end{cases}\)
Option 5: \(\displaystyle R(w) = \lim_{p \rightarrow \infty} \frac{1}{n} \sum_{i = 1}^n |y_i - w|^p\)
a)
b)
(2.5 pts) 11 is the value of \(w\) that minimizes...
Solution
c)
(2.5 pts) 12 is the value of \(w\) that minimizes...
Solution
d)
(2.5 pts) 13 is the value of \(w\) that minimizes...
Solution
e)
(2.5 pts) 17 is the value of \(w\) that minimizes...
Solution
f)
(2.5 pts) 27 is the value of \(w\) that minimizes...
Solution
FA25 MT1 · Problem 2: Absolute Madness 17 pts
From FA25 MT1
Consider a dataset of \(n = 8\) values, where
$$ y_1 = 1,\quad y_2 = y_3 = 4, \quad y_4 = y_5 = y_6 = \alpha,\quad y_7 = y_8 = 20 $$
and \(4 < \alpha < 20\).
As usual, let \(R_\text{abs}(w)\) represent the mean absolute error of a constant prediction \(w\) on this dataset of 8 values.
a)
3 pts Is the value of \(w^{\ast}\), the minimizer of \(R_\text{abs}(w)\), unique? Select and fill out one option below.
The value of \(w^{\ast}\) is unique, and is equal to ______.
The value of \(w^{\ast}\) is not unique; any value between ______ and ______ is a minimizer.
Solution
The value of \(w^{\ast}\) is unique, and is equal to \(\boxed{\alpha}\).
The value of \(w^{\ast}\) is not unique; any value between ______ and ______ is a minimizer.
b)
6 pts Find the value of \(R_\text{abs}(\alpha)\), for any valid choice of \(\alpha\). Show your work, and \(\boxed{\text{circle}}\) your final answer, which should be an expression involving \(\alpha\) and other constants, but no other variables, and no summation notation.
Solution
\(R_\text{abs}(\alpha) = \frac{\alpha + 31}{8}\).
Let’s start with the definition of \(R_\text{abs}(w)\) and plug in \(w = \alpha\).
$$ \begin{align*} R_\text{abs}(\alpha) &= \frac{1}{8} \sum_{i=1}^8 |y_i - \alpha| \\\\ &= \frac{|1 - \alpha| + |4 - \alpha| + |4 - \alpha| + |\alpha - \alpha| + |\alpha - \alpha| + |\alpha - \alpha| + |20 - \alpha| + |20 - \alpha|}{8} \\\\ &= \frac{|1 - \alpha| + 2|4 - \alpha| + 2|20 - \alpha|}{8} \end{align*} $$
Since \(\alpha > 1\) and \(\alpha > 4\), we know that \(|1 - \alpha| = \alpha - 1\) and \(|4 - \alpha| = \alpha - 4\). Similarly, since \(\alpha < 20\), we have \(|20 - \alpha| = 20 - \alpha\).
$$ \begin{align*} R_\text{abs}(\alpha) &= \frac{|1 - \alpha| + 2|4 - \alpha| + 2|20 - \alpha|}{8} \\\\ &= \frac{\alpha - 1 + 2(\alpha - 4) + 2(20 - \alpha)}{8} \\\\ &= \frac{\alpha - 1 + 2\alpha - 8 + 40 - 2\alpha}{8} \\\\ &= \frac{\alpha + 31}{8} \end{align*} $$
Recall,
$$ y_1 = 1,\quad y_2 = y_3 = 4, \quad y_4 = y_5 = y_6 = \alpha,\quad y_7 = y_8 = 20 $$
where \(4 < \alpha < 20\).
c)
8 pts Let the minimum possible value of \(R_\text{abs}(w)\) be \(M\). Given that
$$ R_\text{abs}(20) - M = \frac{9}{2} $$
find the value of \(\alpha\). Show your work, and \(\boxed{\text{circle}}\) your final answer, which should be a number with no variables.
Hint: It’s possible to answer this without using your answer from the previous part.
Solution
\(\alpha = 11\).
Since \(\alpha\) minimizes \(R_\text{abs}(w)\), we know that \(R_\text{abs}(\alpha) = M\). In the previous part, we found an expression for \(R_\text{abs}(\alpha)\). One common solution was to find another expression for \(R_\text{abs}(20)\) (which is also a function of \(\alpha\)), and then to solve for the \(\alpha\) such that
$$ R_\text{abs}(20) - R_\text{abs}(\alpha) = \frac{9}{2} $$
Here’s another solution. Since \((\alpha, M)\) is the vertex of \(R_\text{abs}(w)\), we know that the slope to the left of it is negative and the slope to the right of it is positive.
The slope on the line segment between \((\alpha, M)\) and \((20, M + \frac{9}{2})\) is
$$ \text{slope} = \frac{\# \text{left} - \# \text{right}}{n} = \frac{6 - 2}{8} = \frac{1}{2} $$
So, now we know that on the line segment between \((\alpha, M)\) and \((20, M + \frac{9}{2})\), the slope is \(\frac{1}{2}\). This is all we need to solve for \(\alpha\). Since the slope of a line segment is its change in \(y\) over its change in \(x\), we have:
$$ \frac{M + \frac{9}{2} - M}{20 - \alpha} = \frac{1}{2} $$
Solving for \(\alpha\), we get:
$$ \alpha = 11 $$
FA25 Final · Problem 1 10 pts MT1 Redemption
From FA25 Final
a)
6 pts Suppose we’d like to find the optimal constant prediction, \(w^{\ast}\), for the constant model \(h(x_i) = w\), given a dataset of \(n\) values \(y_1, y_2, \ldots, y_n\). To do so, we minimize mean Bursley error, defined as
$$ R_{\text{B}}(w) = \frac{1}{n} \sum_{i=1}^n | 2y_i - w |^2 $$
Suppose the mean of \(y_1, y_2, \ldots, y_n\) is 20 and the median of \(y_1, y_2, \ldots, y_n\) is 30.
Which value of \(w^{\ast}\) minimizes \(R_{\text{B}}(w)\) for this dataset? Select one of the answers below, then justify your answer in the box provided.
Hint: Look very closely at the definition of \(R_{\text{B}}(w)\). You do not need to re-prove any results from class; you can fully find and explain your answer without using calculus.
Answer:
Justify your answer in the box below.
Solution
First, notice that the use of absolute values is a distraction: since \(|x|^2 = x^2\), we can rewrite \(R_{\text{B}}(w)\) as
$$ R_{\text{B}}(w) = \frac{1}{n} \sum_{i=1}^n (2y_i - w)^2 $$While it’s possible to solve this problem by taking the derivative of \(R_{\text{B}}(w)\) with respect to \(w\) and setting it equal to 0, it’s quicker to leverage what we already know. We know that if there wasn’t a coefficient of \(2\) in front of \(y_i\), the minimizer would be the mean of the dataset.
One way to reason about the effect of the coefficient of \(2\) is to consider a substitution. Let \(z_i = 2y_i\). Then, \(R_{\text{B}}(w)\) becomes
$$ R_{\text{B}}(w) = \frac{1}{n} \sum_{i=1}^n (z_i - w)^2 $$which is the same as the mean squared error of the dataset \(z_1, z_2, \ldots, z_n\), and so \(w^{\ast} = \bar{z}\). But \(\bar{z} = 2 \bar{y}\), and so
$$ w^* = 2 \bar{y} = 2 \cdot 20 = \boxed{40} $$
b)
4 pts This part does not use any of the numbers from part a).
Recall that the mean absolute error, \(R_{\text{abs}}(w)\), of a constant prediction \(w\) on a dataset of \(n\) values \(y_1, y_2, \ldots, y_n\) is given by
$$ R_{\text{abs}}(w) = \frac{1}{n} \sum_{i=1}^n |y_i - w| $$
Consider the dataset of 4 values, \(1, 3, 5, 9\). Among all integers not in this dataset, which integer minimizes \(R_{\text{abs}}(w)\) for this dataset?
\(\text{minimizer} = \_\_\_\_\_\_\)
Solution
The minimizer of mean absolute error is the median of the dataset. When the number of data points is even, any value between the middle two values, inclusive, minimizes mean absolute error. Here, any value between 3 and 5 inclusive minimizes mean absolute error; the value \(\boxed{4}\) is the only integer in this range that isn’t in the dataset itself, so it is the minimizer we’re looking for.
WN26 MT1 · Problem 1 16 pts 🎥 Part c
From WN26 MT1
Consider a dataset of \(n\) values, \(y_1, y_2, \ldots, y_n\), with:
a mean of \(\bar{y} = 18\)
a median of 15
a standard deviation of \(\sigma_y = 7\)
a)
4 pts In the space provided, sketch the graph of \(R_\text{sq}(w)\), the mean squared error of a constant prediction \(w\) on the dataset. For full credit:
The shape of the graph must be correct.
You must clearly label the coordinates of the minimum point on the graph.
Solution
Recall that
$$ R_\text{sq}(w) = \frac{1}{n} \sum_{i=1}^n (y_i - w)^2 $$
is a parabola, minimized at \(w = \bar y\). When \(w = \bar y\),
$$ R_\text{sq}(w) = \frac{1}{n} \sum_{i=1}^n (y_i - \bar y)^2 = \sigma_y^2 $$
is the variance of the dataset. Here, the mean is 18 and the variance is 49, so the minimum point is at \((18, 49)\).
b)
6 pts Which of the following quantities is guaranteed to be equal to 0? Select all that apply.
\(\displaystyle \frac{1}{n} \sum_{i=1}^n (y_i - 15)\)
\(\displaystyle \frac{1}{n} \sum_{i=1}^n (y_i - 18)\)
\(\displaystyle \frac{1}{n}\sum_{i=1}^n (y_i - 15)^2\)
\(\displaystyle \frac{1}{n}\sum_{i=1}^n (y_i - 18)^2\)
\(\displaystyle \frac{1}{n}\sum_{i=1}^n (y_i - 15)^2 - 7^2\)
\(\displaystyle \frac{1}{n}\sum_{i=1}^n (y_i - 18)^2 - 7^2\)
Solution
\(\displaystyle \frac{1}{n}\sum_{i=1}^n (y_i - 18)^2 - 7^2\)
There are two key ideas at play here:
- The mean is the unique point in the dataset such that the sum of deviations from the mean is 0. In other words,
$$ \sum_{i=1}^n (y_i - \bar y) = \sum_{i=1}^n y_i - n \bar y = n \bar y - n \bar y = 0 $$
- The variance of a dataset is the average of the squared deviations from the mean. In other words,
$$ \sigma_y^2 = \frac{1}{n} \sum_{i=1}^n (y_i - \bar y)^2 $$
Equivalently, this is the value of \(R_\text{sq}(w)\) when \(w = \bar y\).
With this in mind, let’s look at the options:
(False) \(\displaystyle \frac{1}{n} \sum_{i=1}^n (y_i - 15)\): This is the average of the deviations from the median, which is not 0. This is only true for the mean.
(True) \(\displaystyle \frac{1}{n} \sum_{i=1}^n (y_i - 18)\): This is the average of the deviations from the mean, which is 0. This is only true for the mean.
(False) \(\displaystyle \frac{1}{n}\sum_{i=1}^n (y_i - 15)^2\): This is the function \(R_\text{sq}(w)\) when \(w = 15\). As we see in the solution to part a), this is not 0.
(False) \(\displaystyle \frac{1}{n}\sum_{i=1}^n (y_i - 18)^2\): This is the function \(R_\text{sq}(w)\) when \(w = 18\), i.e. it is the variance of the dataset. As we see in the solution to part a), this is also not zero — here, it is \(\sigma_y^2 = 7^2 = 49\). One point of confusion may be that \(w = \bar{y}\) is the point at which \(R_\text{sq}(w)\) is minimized and \(R_\text{sq}(w)\) has a derivative of 0, but \(R_\text{sq}(\bar y) \neq 0\) in general.
(False) \(\displaystyle \frac{1}{n}\sum_{i=1}^n (y_i - 15)^2 - 7^2\): This would be true if the 15 were replaced with the mean, 18, but it is not.
(True) \(\displaystyle \frac{1}{n}\sum_{i=1}^n (y_i - 18)^2 - 7^2\): This is the variance of the dataset minus the variance of the dataset, which indeed is 0.
c)
6 pts Recall that \(R_\text{abs}(w)\) is the mean absolute error of a constant prediction \(w\) on the dataset. A snippet of the graph of \(R_\text{abs}(w)\) is shown below.
For clarity, the circles at \((15, 4)\), \((18, 5)\), and \((22, 7)\) indicate the points at which the slope of \(R_\text{abs}(w)\) changes.
Given that there are \(n = 72\) values in the dataset, how many values in the dataset are equal to 18? Show your work and \(\boxed{\text{circle}}\) your final answer, which should be an integer with no variables.
Solution
The number of values in the dataset that are equal to 18 is 6.
Recall, the slope of \(R_\text{abs}(w)\) at any \(w\) that is not a data point is:
$$ \frac{\text{d}}{\text{d}w} R_\text{abs}(w) = \frac{\# \text{ left of } w - \# \text{ right of } w}{n} $$
There are two line segments of interest here: the one between \(w=15\) and \(w=18\), and the one between \(w=18\) and \(w=22\). We have two ways of computing the slope of each one: by using \(\text{slope} = \frac{\text{change in } y}{\text{change in } x}\) and by using the formula above. We’ll use both formulas on each line segment.
Between \(w=15\) and \(w=18\):
Method 1: Using \(\text{slope} = \frac{\text{change in } y}{\text{change in } x}\), the graph rises from \((15, 4)\) to \((18, 5)\), which gives a slope of
$$ s_1 = \frac{5 - 4}{18 - 15} = \frac{1}{3} $$
- Method 2: Using the formula for the slope of \(R_\text{abs}(w)\), let \(l\) be the number of values in the dataset less than or equal to 15. Then, the slope in this interval is
$$ s_1 = \frac{l - (72 - l)}{72} = \frac{2l - 72}{72} $$
At this point, we have enough information to solve for \(l\):
$$ \frac{2l - 72}{72} = \frac{1}{3} \implies l = 48 $$
Between \(w=18\) and \(w=22\):
Method 1:
$$ s_2 = \frac{7 - 5}{22 - 18} = \frac{2}{4} = \frac{1}{2} $$
- Method 2: Let \(k\) be the number of values in the dataset equal to 18. Ultimately, this is what we’re trying to find. Then, the number of values in the dataset less than or equal to 18 is \(l + k\). In this interval, the slope is
$$ s_2 = \frac{(l + k) - (72 - (l + k))}{72} = \frac{2(l + k) - 72}{72} $$
So, we need to solve for \(k\) in
$$ \frac{2(l + k) - 72}{72} $$
But, we know that \(l = 48\), so
$$ \frac{2(48 + k) - 72}{72} = \frac{1}{2} \implies 96 + 2k - 72 = 36 \implies 2k = 12 \implies \boxed{k = 6} $$
Therefore, there are 6 values in the dataset that are equal to 18.
WN26 MT1 · Problem 7 20 pts 🎥 Part c
From WN26 MT1
Suppose we’d like to find the optimal constant parameter, \(w^{\ast}\), for the constant model \(h(x_i) = w\), given a dataset of \(n\) points \((x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)\). To do so, we use the sub-squared loss function, \(L_\text{ss}\), defined below.
$$ L_\text{ss}(y_i, w) = (\sqrt{y_i} - \sqrt{w})^2 $$
This requires us to assume that all \(y_i \ge 0\), as are all possible values of \(w\).
a)
6 pts Find \(\frac{\text{d}}{\text{d}w} R_\text{ss}(w)\), the derivative of average sub-squared loss (i.e. the empirical risk) with respect to \(w\). Show your work and \(\boxed{\text{circle}}\) your final answer, which should be an expression in terms of the \(y_i\)’s, \(n\), and/or any constants. Hint: The derivative of \(f(x) = \sqrt{x}\) is \(\frac{\text{d}}{\text{d}x} \sqrt{x} = \frac{1}{2\sqrt{x}}\).
Solution
The definition of average sub-squared loss is
$$ R_\text{ss}(w) = \frac{1}{n} \sum_{i=1}^n L_\text{ss}(y_i, w) = \frac{1}{n} \sum_{i=1}^n (\sqrt{y_i} - \sqrt{w})^2 $$
Then,
$$ \begin{align*} \frac{\text{d}}{\text{d}w} R_\text{ss}(w) &= \frac{\text{d}}{\text{d}w} \left( \frac{1}{n} \sum_{i=1}^n (\sqrt{y_i} - \sqrt{w})^2 \right) \\\\ &= \frac{1}{n} \sum_{i=1}^n \frac{\text{d}}{\text{d}w} \left[(\sqrt{y_i} - \sqrt{w})^2\right] \\\\ &= \frac{1}{n} \sum_{i=1}^n 2(\sqrt{y_i} - \sqrt{w}) \cdot \frac{\text{d}}{\text{d}w} (\sqrt{y_i} - \sqrt{w}) \\\\ &= \frac{1}{n} \sum_{i=1}^n 2(\sqrt{y_i} - \sqrt{w}) \cdot \left(0 -\frac{1}{2\sqrt{w}} \right) \\\\ &= \boxed{-\frac{1}{n} \sum_{i=1}^n \frac{\sqrt{y_i} - \sqrt{w}}{\sqrt{w}}} \end{align*} $$
b)
6 pts Show that the value of \(w^{\ast}\) that minimizes average sub-squared loss is
$$ \displaystyle w^* = \left( \frac{1}{n} \sum_{i=1}^n \sqrt{y_i} \right)^2 $$
Solution
We’ve found that
$$ \frac{\text{d}}{\text{d}w} R_\text{ss}(w) = -\frac{1}{n} \sum_{i=1}^n \frac{\sqrt{y_i} - \sqrt{w}}{\sqrt{w}} $$
To find \(w^{\ast}\), we need to set this expression equal to 0 and solve for \(w\).
$$ \begin{align*} -\frac{1}{n} \sum_{i=1}^n \frac{\sqrt{y_i} - \sqrt{w}}{\sqrt{w}} = 0 \\\\ \sum_{i=1}^n \frac{\sqrt{y_i} - \sqrt{w}}{\sqrt{w}} = 0 \\\\ \sum_{i=1}^n (\sqrt{y_i} - \sqrt{w}) = 0 \\\\ \sum_{i=1}^n \sqrt{y_i} - n \sqrt{w} = 0 \\\\ \sqrt{w} = \frac{1}{n} \sum_{i=1}^n \sqrt{y_i} \\\\ w = \left( \frac{1}{n} \sum_{i=1}^n \sqrt{y_i} \right)^2 \end{align*} $$
So, the value of \(w^{\ast}\) that minimizes average sub-squared loss is
$$ \boxed{w^* = \left( \frac{1}{n} \sum_{i=1}^n \sqrt{y_i} \right)^2} $$
c)
6 pts Using the Cauchy-Schwarz inequality, prove that
$$ \left( \frac{1}{n} \sum_{i=1}^n \sqrt{y_i} \right)^2 \leq \bar{y} $$
where \(\bar{y}\) is the mean of the \(y_i\)’s.
Solutions that do not use the Cauchy-Schwarz inequality will not receive credit.
Solution
The Cauchy-Schwarz inequality states that
$$ \left| \vec u \cdot \vec v \right| \leq \lVert \vec u \rVert \lVert \vec v \rVert $$
for any two vectors \(\vec u, \vec v \in \mathbb{R}^n\). The problem boils down to constructing \(\vec u\) and \(\vec v\) such that the Cauchy-Schwarz inequality, for that \(\vec u\) and \(\vec v\), implies the inequality we’re trying to prove.
For hints on how to proceed, let’s expand the definition of \(\bar y\) in the inequality we’re trying to prove.
$$ \left( \frac{1}{n} \sum_{i=1}^n \sqrt{y_i} \right)^2 \leq \frac{1}{n} \sum_{i=1}^n y_i $$
On the left, we have a sum of \(\sqrt{y_i}\)’s, and on the right, we have a sum of \(y_i\)’s. We know that in the norm of a vector, the individual components are squared, which would allow us to turn \(\sqrt{y_i}\) into \(y_i\). So, one possible path forward is
$$ \vec u = \begin{bmatrix} \sqrt{y_1} \\\\ \sqrt{y_2} \\\\ \vdots \\\\ \sqrt{y_n} \end{bmatrix}, \qquad \vec v = \begin{bmatrix} 1 \\\\ 1 \\\\ \vdots \\\\ 1 \end{bmatrix} $$
The dot product of \(\vec u\) and \(\vec v\) is \(\sum_{i=1}^n \sqrt{y_i}\), which seems promising. Let’s plug \(\vec u\) and \(\vec v\) into the Cauchy-Schwarz inequality and see what we get.
$$ \begin{align*} \left| \vec u \cdot \vec v \right| &\leq \lVert \vec u \rVert \lVert \vec v \rVert \\\\ \left| \sum_{i=1}^n \sqrt{y_i} \right| &\leq \left \lVert \begin{bmatrix} \sqrt{y_1} \\\\ \sqrt{y_2} \\\\ \vdots \\\\ \sqrt{y_n} \end{bmatrix} \right \rVert \left \lVert \begin{bmatrix} 1 \\\\ 1 \\\\ \vdots \\\\ 1 \end{bmatrix} \right \rVert \\\\ \sum_{i=1}^n \sqrt{y_i} &\leq \sqrt{\sum_{i=1}^n y_i} \sqrt{n} \\\\ \end{align*} $$
Seems like we’re getting somewhere. Let’s square both sides.
$$ \begin{align*} \left( \sum_{i=1}^n \sqrt{y_i} \right)^2 &\leq \left( \sqrt{\sum_{i=1}^n y_i} \sqrt{n} \right)^2 \\\\ \left( \sum_{i=1}^n \sqrt{y_i} \right)^2 & \leq n\sum_{i=1}^n y_i \end{align*} $$
Now, all that’s left is to divide both sides by \(n^2\).
$$ \begin{align*} \left( \sum_{i=1}^n \sqrt{y_i} \right)^2 & \leq n\sum_{i=1}^n y_i \\\\ \frac{\left( \sum_{i=1}^n \sqrt{y_i} \right)^2}{n^2} & \leq \frac{n\sum_{i=1}^n y_i}{n^2} \\\\ \left( \frac{1}{n} \sum_{i=1}^n \sqrt{y_i} \right)^2 & \leq \frac{1}{n} \sum_{i=1}^n y_i \end{align*} $$
This is exactly the inequality we’re trying to prove, so we’re done!
d)
2 pts What is the value of \(w\) that minimizes the following function:
$$ R(w) = \frac{1}{n}\sum_{i=1}^n (y_i^4 - w^4)^2 $$
Hint: This can be done without using any calculus — don’t try and take the derivative.
Solution
The idea here is to make a substitution that reduces the problem to one we’ve already seen — the problem of minimizing mean squared error for the constant model.
Let \(z_i = y_i^4\), and let \(t = w^4\). Then,
$$ \frac{1}{n}\sum_{i=1}^n (z_i - t)^2 = \frac{1}{n}\sum_{i=1}^n (y_i^4 - w^4)^2 $$
What is \(t^{\ast}\), the minimizer of \(\frac{1}{n}\sum_{i=1}^n (z_i - t)^2\)? That’s \(\bar{z}\), which is
$$ t^* = \bar{z} = \frac{1}{n} \sum_{i=1}^n z_i = \frac{1}{n} \sum_{i=1}^n y_i^4 $$
But, \(t = w^4\), so \(w = t^{1/4}\), meaning
$$ w^* = \boxed{\left( \frac{1}{n} \sum_{i=1}^n y_i^4 \right)^{1/4}} $$
Notice how this relates to parts a) and b) — those could have been solved the same way, if you wrote \(\sqrt{x}\) as \(x^{1/2}\).
WN26 Final · Problem 1 12 pts MT1 Redemption
From WN26 Final
Suppose we’d like to find the optimal constant prediction, \(w^{\ast}\), for the constant model \(h(x_i) = w\), given the following dataset of \(n = 4\) values.
$$ y_1 = 3, \quad y_2 = 6, \quad y_3 = 6, \quad y_4 = 13 $$
In each part, choose from the options below.
$$ \begin{array}{l@{\hspace{1.75cm}}l} A = 3 & E = 7 \\\\[1.5ex] B = \dfrac{4}{\frac{1}{3} + \frac{1}{6} + \frac{1}{6} + \frac{1}{13}} \approx 5.37 & F = \sqrt{\dfrac{3^2 + 6^2 + 6^2 + 13^2}{4}} \approx 7.90 \\\\[3ex] C = 6 & G = 8 \\\\[1.5ex] D = \left( 3 \cdot 6 \cdot 6 \cdot 13 \right)^{1/4} \approx 6.12 & H = 13 \\\\ \end{array} $$
(3 pts) What value of \(w^{\ast}\) minimizes \(R(w) = \displaystyle \frac{1}{4} \sum_{i=1}^4 (y_i - w)^2\)?
Solution
For (i), the minimizer of mean squared error is the mean, so
$$ w^* = \frac{3+6+6+13}{4} = \boxed{7} $$(3 pts) What value of \(w^{\ast}\) minimizes \(R(w) = \displaystyle \lim_{p \to \infty} \displaystyle \frac{1}{4} \sum_{i=1}^4 |y_i - w|^p\)?
Solution
For (ii), as \(p \to \infty\), the largest value of \(|y_i-w|\) dominates. So we should put \(w\) halfway between the smallest and largest data values, as discussed in Chapter 1.4.
$$ w^* = \frac{3+13}{2} = \boxed{8} $$(3 pts) What value of \(w^{\ast}\) minimizes \(R(w) = \displaystyle \frac{1}{4} \sum_{i=1}^4 (\log(y_i) - \log(w))^2\)?
Solution
For (iii), let \(u=\log(w)\). The problem is now asking for the best constant prediction for the transformed values \(\log(y_i)\), so
$$ u^* = \frac{\log(3)+\log(6)+\log(6)+\log(13) = \log(3 \cdot 6 \cdot 6 \cdot 13)}{4} $$Exponentiating gives
$$ w^* = e^{u^*} = \boxed{(3 \cdot 6 \cdot 6 \cdot 13)^{1/4}} $$This was also a homework problem.
(3 pts) The slope of the graph of \(R(w) = \displaystyle\frac{1}{4} \sum_{i = 1}^4 |y_i - w|\) at \(w = \alpha\) is \(-1/2\). Among the options above, which could be \(\alpha\)?
Solution
For (iv), the slope of mean absolute error at any \(w\) that is not a data point is
$$ \frac{\text{\# left of } w - \text{\# right of } w}{n} $$Here, in order to achieve a slope of \(-1/2\), we need to have 1 data point to the left of \(w\) and 3 to the right, since \(\frac{1-3}{4} = -1/2\). This means we need \(w\) to be between \(3\) and \(6\), exclusive. The only value in this interval is \(B\),
$$ \boxed{\dfrac{4}{\frac{1}{3}+\frac{1}{6}+\frac{1}{6}+\frac{1}{13}} \approx 5.37} $$
SP26 MT1 · Problem 1 16 pts
From SP26 MT1
Suppose we’d like to find the optimal parameter, \(w^{\ast}\), for the constant model \(h(x_i)=w\), using the following dataset of \(n = 4\) values, \(y_1, y_2, y_3, y_4\):
$$ 0, \quad 2, \quad 4, \quad 20 $$
a)
3 pts First, suppose we find the optimal parameter by minimizing mean squared error, \(R_\text{sq}(w)\). Which value of \(w\) minimizes \(R_\text{sq}(w)\)? Give your answer as a number with no variables.
\(\text{minimizer of } R_\text{sq}(w) = \_\_\_\_\_\_\)
Solution
For the constant model, average squared loss is minimized at the mean of the \(y_i\)’s. Here,
$$ \frac{0+2+4+20}{4}=\frac{26}{4}=\boxed{\frac{13}{2}} $$
Now, consider the clipped loss function, defined below.
$$ \displaystyle L_\text{clip}(y_i,h(x_i))=\min\{(y_i-h(x_i))^2,9\} $$
For example, \(L_\text{clip}(10, 5) = 9\) and \(L_\text{clip}(5, 3) = 4\).
Let \(R_\text{clip}(w)\) be the average clipped loss for the constant model and this dataset.
b)
3 pts State one value of \(w\) where the derivative of \(R_\text{clip}(w)\) is not defined.
\(\text{one value of } w \text{ where the derivative of } R_\text{clip}(w) \text{ is not defined} =\) ______
Solution
The clipped loss changes formulas whenever
$$ (y_i-w)^2=9 $$
Equivalently, this happens when \(w=y_i\pm 3\). Since \(20-3=17\), one valid answer is \(\boxed{17}\).
For context, here’s what average clipped loss looks like for this dataset:
c)
3 pts Suppose we restrict \(w\) to the interval \(1 \leq w \leq 3\). Among all values of \(w\) in this interval, which value minimizes \(R_\text{clip}(w)\)? Give your answer as a number with no variables.
\(\text{minimizer of } R_\text{clip}(w) \text{ within the interval } [1, 3] = \_\_\_\_\_\_\)
Solution
Once \(w\) is more than 3 units away from any particular \(y_i\) value, the value \((y_i - w)^2\) is replaced by the constant \(9\) when computing average loss.
What do we know about constants when they are added to functions? They don’t affect the minimizer! That is, the minimizer of \(f(x)\) and of \(f(x) + c\) are the same.
What this is saying is that if \(w\) is restricted to the interval \(1 \leq w \leq 3\), we can ignore \(y_4 = 20\) when computing the minimizer, and this just reduces to minimizing average squared loss (mean squared error) across the data points that are within 3 units of \(w\). As long as \(1 \leq w \leq 3\), we are within 3 units of \(y_1 = 0\), \(y_2 = 2\), and \(y_3 = 4\).
What constant minimizes average squared loss, for the dataset \(0, 2, 4\)? That’s the mean of \(0, 2, 4\), which is \(2\). So the minimizer of \(R_\text{clip}(w)\) within the interval \(1 \leq w \leq 3\) is \(\boxed{2}\).
If you’d like to see this a little more formally, then when \(1 \leq w \leq 3\),
$$ R_\text{clip}(w)=\frac14\left(w^2+(2-w)^2+(4-w)^2+9\right) $$
Taking the derivative,
$$ \frac{\text{d}}{\text{d}w}R_\text{clip}(w)=\frac14(2w+2(w-2)+2(w-4))=\frac{6w-12}{4} $$
Setting this equal to \(0\) gives \(w = 2\), as we intuited earlier.
d)
3 pts Now suppose there are no restrictions on \(w\). Among all possible values of \(w\), which value minimizes \(R_\text{clip}(w)\)? Give your answer as a number with no variables.
\(\text{minimizer of } R_\text{clip}(w) = \_\_\_\_\_\_\)
Solution
The best \(w\) is still \(w = 2\). As a refresher, let’s look at the graph of \(R_\text{clip}(w)\) again:
First, note that \(w = 20\) is a local minimizer of \(R_\text{clip}(w)\): if we zoom in to the graph of \(R_\text{clip}(w)\) around \(w = 20\), it looks like a parabola that opens up, centered at \(w = 20\). But, when we zoom out, we see that the graph falls even lower near \(w = 2\) than it does near \(w = 20\).
Why is this? It’s because there are many more \(y_i\) values within 3 units of \(w = 2\) than there are within 3 units of \(w = 20\). Remembering that we have \(y_1 = 0, y_2 = 2, y_3 = 4, y_4 = 20\):
$$ R_\text{clip}(20) = \frac{1}{4} \sum_{i=1}^4 \min\{(20-y_i)^2, 9\} = \frac{1}{4} \left( 9 + 9 + 9 + 0 \right) = \frac{27}{4} $$
$$ R_\text{clip}(2) = \frac{1}{4} \sum_{i=1}^4 \min\{(2-y_i)^2, 9\} = \frac{1}{4} \left( 4 + 0 + 4 + 9 \right) = \frac{17}{4} $$
So, \(R_\text{clip}(20) = \frac{27}{4} > \frac{13}{4} = R_\text{clip}(2)\).
The question, then, is whether \(w=2\) is the global minimizer, or just that it’s better than \(w=20\). Crucially, you wouldn’t have had the graph of \(R_\text{clip}(w)\) during the exam, so you would have needed to reason about this without it. One way to see how \(w = 2\) is the global minimizer is to realize that as \(w\) increases from \(2\), the average loss only increases, until it reaches 9, where it “coasts” until it we reach \(w = 17\), where it decreases once again.
e)
4 pts State one pro and one con of using clipped loss instead of squared loss to find optimal model parameters.
Solution
One pro is that clipped loss is less sensitive to outliers, since very large errors all receive the same loss of \(9\). One con is that it stops distinguishing between bad and very bad predictions once the error is large enough; it also introduces points where the derivative is not defined, when the two cases of the min function switch.
SP26 MT1 · Problem 2 10 pts
From SP26 MT1
We will continue to use the constant model, \(h(x_i)=w\), and the same dataset of \(n=4\) values as in Problem 1:
$$ 0,\quad 2,\quad 4,\quad 20 $$
Instead of the clipped loss function, consider the weighted absolute loss function, defined below.
$$ L_\text{WA}(y_i,h(x_i))= \begin{cases} \beta(y_i-h(x_i)), & h(x_i)<y_i \\\\ h(x_i)-y_i, & h(x_i)\ge y_i \end{cases} $$
where \(\beta\) is a positive integer. Let \(R_\text{WA}(w)\) be the average weighted absolute loss for the constant model and this dataset.
The slope of \(R_\text{WA}(w)\) at \(w\), for any value of \(w\) not equal to one of the \(y_i\) values, is
$$ \text{slope of } R_\text{WA}(w) \text{ at } w = \frac{\#\text{ left of } w - \beta(\#\text{ right of } w)}{4} $$
a)
4 pts Suppose \(\beta = 1\). Which value of \(w\) minimizes \(R_\text{WA}(w)\)? Show your work, and write your final answer in the box provided. Your answer should be a number with no variables. If there are multiple possible answers, state just one.
$$ \text{minimizer of } R_\text{WA}(w) = \_\_\_\_\_\_ $$
Solution
When \(\beta = 1\), \(R_{\text{WA}}(w)\) is just mean absolute error, \(R_\text{abs}(w)\). We know that the minimizer of mean absolute error is the median of the dataset, or any value between the middle two values if the dataset has an even number of values.
This dataset has an even number of values, so any \(w\) in the interval \(2 \leq w \leq 4\) minimizes \(R_\text{WA}(w)\). One such value is \(\boxed{3}\), but \(2\), \(4\), \(\pi\), etc. are all valid answers.
b)
6 pts Now suppose \(\beta = 2\). Which value of \(w\) minimizes \(R_\text{WA}(w)\)? Show your work, and write your final answer in the box provided. Your answer should be a number with no variables. If there are multiple possible answers, state just one.
$$ \text{minimizer of } R_\text{WA}(w) = \_\_\_\_\_\_ $$
Solution
When \(\beta=2\), the slopes between consecutive data values are
$$ -2,\quad -\frac54,\quad -\frac12,\quad \frac14,\quad 1 $$
on the intervals \((-\infty,0)\), \((0,2)\), \((2,4)\), \((4,20)\), and \((20,\infty)\). The slope changes from negative to positive at \(w=4\), so the minimizer is \(\boxed{4}\).
Conceptually, the fact that the errors in the case where \(y_i > h(x_i)\) are multiplied by \(\beta\) forces the optimal \(w^{\ast}\) to be larger than the median (since we want the \(y_i > h(x_i)\) case to not trigger as often when computing the average loss across the entire dataset).
SP26 Final · Problem 1 14 pts MT1 Redemption
From SP26 Final
Suppose we’d like to find the optimal constant parameter, \(w^{\ast}\), for the constant model \(h(x_i)=w\), using the following dataset of \(n=5\) values:
$$ 1,\quad 1,\quad 4,\quad 9,\quad 25 $$
In each part, find the value of \(w^{\ast}\) that minimizes the given \(R(w)\). Show your work in the space provided, and write your final answer in the bottom-right corner of the box. Your answers should be numbers with no variables. Note: There is no need to use calculus here.
a)
4 pts \(\displaystyle R(w) = \frac{1}{5} \sum_{i=1}^5 (y_i - w)^2\)
$$ w^* = \_\_\_\_\_\_ $$
Solution
The minimizer of mean squared error for a constant model is the mean, as discussed in Chapter 1.2. So,
$$ w^* = \frac{1+1+4+9+25}{5} = \frac{40}{5} = 8 $$
b)
4 pts \(\displaystyle R(w) = \frac{1}{5} \sum_{i=1}^5 (\sqrt{y_i} - w)^2\)
$$ w^* = \_\_\_\_\_\_ $$
Solution
This is asking for the best constant prediction for the transformed values \(\sqrt{y_i}\). The transformed data are
$$ 1,\quad 1,\quad 2,\quad 3,\quad 5 $$
so
$$ w^* = \frac{1+1+2+3+5}{5} = \frac{12}{5} $$
c)
4 pts \(\displaystyle R(w) = \frac{1}{5} \sum_{i=1}^5 (y_i - \sqrt{w})^2\)
$$ w^* = \_\_\_\_\_\_ $$
Solution
Let \(u=\sqrt{w}\). The loss becomes
$$ R(u) = \frac{1}{5}\sum_{i=1}^5 (y_i-u)^2 $$
which is minimized at the mean of the original \(y_i\) values:
$$ u^* = \frac{1+1+4+9+25}{5} = 8 $$
Since \(u=\sqrt{w}\), we have
$$ w^* = 8^2 = 64 $$
d)
2 pts Which answer from above is also the minimizer of \(\displaystyle R(w) = \sqrt{\frac{1}{5} \sum_{i=1}^5 (y_i - w)^2}\)?
Solution
The square root function is strictly increasing, so minimizing
$$ \sqrt{\frac{1}{5} \sum_{i=1}^5 (y_i-w)^2} $$
is equivalent to minimizing
$$ \frac{1}{5} \sum_{i=1}^5 (y_i-w)^2 $$
That is exactly the objective from part a), so the answer is the answer from part a).