题目
描述
- $y_i=x_i^Teta+epsilon_i$
$epsilon_isim N(0,sigma^2)$ -
已有训练集$ au$，其中$X:n imes p,y:n imes 1,epsilon:n imes 1$
使用最小二乘得到$hat{eta}=left(X^TX ight)^{-1}X^Ty$
$y=Xeta+epsilon$ - 需要预测点$x_0$的値$y_0$
题2.7
-
准备
- $E(y_0)=E(x_0^Teta+epsilon_0)=E(x_0^Teta)+E(epsilon_0)=x_0^Teta+0$
- $E[left(y_0-E(y_0) ight)^2]=E[left(x_0^Teta+epsilon_0-x_0^Teta ight)^2]=E[epsilon_0^2]=sigma^2$
- $hat{y_0}=x_0^That{eta}=x_0^Tleft(X^TX ight)^{-1}X^Tleft(Xeta+epsilon ight)\ =x_0^Tleft(X^TX ight)^{-1}X^TXeta+x_0^Tleft(X^TX ight)^{-1}X^Tepsilon\ =x_0^Teta+x_0^Tleft(X^TX ight)^{-1}X^Tepsilon\ =x_0^Teta+sum_i a_iepsilon_i$ <br>其中$a_i=left[x_0^Tleft(X^TX ight)^{-1}X^T ight]_i$
- $E(hat{y_0})=E(x_0^Teta+sum_i a_iepsilon_i)=x_0^Teta+sum_i E(a_i)E(epsilon_i)=x_0^Teta$
这里由于$X$由某分布产生，所以$E(a_i)$不是简单常数
-
题解
$EPE(x_0)=intintleft(y_0-hat{y_0} ight)^2p(y_0)p(hat{y_0})mathrm{d} y_0mathrm{d}hat{y_0}\ =intintleft[hat{y_0}-E(hat{y_0})+E(hat{y_0})-y_0 ight]^2p(y_0)p(hat{y_0})mathrm{d} y_0mathrm{d}hat{y_0}\ =intleft[hat{y_0}-E(hat{y_0}) ight]^2p(hat{y_0})mathrm{d}hat{y_0}+intintleft[E(hat{y_0})-y_0 ight]^2p(y_0)p(hat{y_0})mathrm{d} y_0mathrm{d}hat{y_0}+2 imes 0\ ={Var}_ au(hat{y_0})+intintleft[y_0-E(y_0)+E(y_0)-E(hat{y_0}) ight]^2p(y_0)p(hat{y_0})mathrm{d} y_0mathrm{d}hat{y_0}\ ={Var}_ au(hat{y_0})+intleft[y_0-E(y_0) ight]^2p(y_0)mathrm{d} y_0+left[E(y_0)-E(hat{y_0} ight]^2+2 imes 0\ ={Var}_ au(hat{y_0})+sigma^2+0^2$
${Var}_ au(hat{y_0})=Eleft[hat{y_0}-E(hat{y_0}) ight]^2\ =Eleft[x_0^Teta+sum_i a_iepsilon_i-x_0^Teta ight]^2=Eleft[sum_isum_j a_ia_jepsilon_iepsilon_j ight]\ =Eleft[ sum_i a_i^2epsilon_i^2 ight]+Eleft[sum_isum_{j:j eq i} a_ia_jepsilon_iepsilon_j ight]\ =sum_iE(a_i^2)E(epsilon_i^2)+sum_isum_{j:j eq i} E(a_ia_j)E(epsilon_i)E(epsilon_j)\ =sigma^2E(sum_i a_i^2)+0=sigma^2Eleft(x_0^Tleft(X^TX ight)^{-1}X^TXleft(X^TX ight)^{-1}x_0 ight)\ =sigma^2Eleft(x_0^Tleft(X^TX ight)^{-1}x_0 ight)$
题2.8
-
准备
- 假设$E(x^{(i)})=0,i=1...p$，即每个维度的期望都为0
$X^TX$得到$p imes p$的矩阵
$X_{:i}$表示$X$的第$i$列，即训练集输入部分的第i个维度
$X_{:i}^TX_{:i}=sum_j^N {x_j^{(i)}}^2=N frac{1}{N}sum_j^N (x_j^{(i)}-E(x^{(i)}))^2=Nhat{Var}(x^{(i)})$得到对角元素
$X_{:i}^TX_{:j}=sum_t^N x_t^{(i)}x_t^{(j)} = N frac{1}{N} (x_t^{(i)}-E(x^{(i)}))(x_t^{(j)}-E(x^{(j)}))=Nhat{Cov}(x^{(i)},x^{(j)})$
当$N o infty $，$X^TX o NCov(x)$ - $K:p imes p,b:p imes 1$
$trace Kbb^T=sum_i {[Kbb^T]}_{ii}=sum_i sum_j K_{ij}{[bb^T]}_{ji}=sum_i sum_j K_{ij}b_ib_j$
$b^TKb=sum_i {b^T}_{1i}{[Kb]}_{i1}=sum_i sum_j b_iK_{ij}b_j$
$trace Kbb^T=b^TKb$
- 假设$E(x^{(i)})=0,i=1...p$，即每个维度的期望都为0
-
题解
$Eleft(x_0^Tleft(X^TX ight)^{-1}x_0 ight)sim Eleft(x_0^T{Cov(x)}^{-1}x_0 ight)/N\ =Eleft(trace {Cov(x)}^{-1}x_0x_0^T ight)/N\ =trace {Cov(x)}^{-1}E(x_0x_0^T)/N=trace {Cov(x)}^{-1}Cov(x)/N\ =trace I/N=p/N$
$EPE(x_0)=(p/N+1)sigma^2$