CS229 斯坦福大学机器学习复习材料(数学基础) - 线性代数
线性代数提供了一种紧凑地表示和运算“线性方程组”的方法。例如，考虑以下方程组：
4 x 1 − 5 x 2 = − 13 − 2 x 1 + 3 x 2 = 9. 4x_{scriptscriptstyle 1} - 5x_{scriptscriptstyle 2} = -13\
-2x_{scriptscriptstyle 1} + 3x_{scriptscriptstyle 2} = 9. 4 x 1 − 5 x 2 = − 13 − 2 x 1 + 3 x 2 = 9.
这是两个方程和两个变量，正如你从高中代数中所知，你可以找到 x 1 x_1 x 1 x 2 x_2 x 2
A x = b Ax= b A x = b
其中
A = [ 4 − 5 − 2 3 ] , b = [ − 13 9 ] . A=egin{bmatrix}
4 & -5 \
-2 & 3
end{bmatrix},
b=egin{bmatrix}
-13 \
9
end{bmatrix}. A = [ 4 − 2 − 5 3 ] , b = [ − 13 9 ] .
我们可以看到，以这种形式分析线性方程有许多优点（包括明显的节省空间）。
我们使用以下符号：
用A ∈ R m × n A in Bbb{R}^{m imes n} A ∈ R m × n m m m n n n A A A
用x ∈ R n oldsymbol{x} in Bbb{R}^{n} x ∈ R n n n n n n n n n n 1 1 1 列向量 。如果我们想表示一个行向量 : 具有 1 1 1 n n n x T oldsymbol{x}^T x T x T oldsymbol{x}^T x T x oldsymbol{x} x
用x i x_i x i x oldsymbol{x} x i i i x = [ x 1 x 2 ⋮ x n ] . oldsymbol{x}=egin{bmatrix}
x_1 \
x_2 \
vdots \
x_n
end{bmatrix}. x = ⎣ ⎡ x 1 x 2 ⋮ x n ⎦ ⎤ .
我们用符号a i j a_{scriptscriptstyle ij} a ij A i j A_{ij} A ij A i , j A_{i,j} A i , j A A A i i i j j j
A = [ a 11 a 12 ⋯ a 1 n a 21 a 22 ⋯ a 2 n ⋮ ⋮ ⋱ ⋮ a m 1 a m 2 ⋯ a m n ] . A=egin{bmatrix}
a_{11} & a_{12} & cdots & a_{1n} \
a_{21} & a_{22} & cdots & a_{2n} \
vdots & vdots & ddots & vdots \
a_{m1} & a_{m2} & cdots & a_{mn}
end{bmatrix}. A = ⎣ ⎡ a 11 a 21 ⋮ a m 1 a 12 a 22 ⋮ a m 2 ⋯ ⋯ ⋱ ⋯ a 1 n a 2 n ⋮ a mn ⎦ ⎤ .
我们将A A A j j j a j a^j a j A : , j A_{:,j} A : , j
A = [ | | | a 1 a 2 ⋯ a n | | | ] . A = egin{bmatrix}
ext{ extbar} & ext{ extbar} & & ext{ extbar} \
a^1 & a^2 & cdots & a^n \
ext{ extbar} & ext{ extbar} & & ext{ extbar}
end{bmatrix}. A = ⎣ ⎡ | a 1 | | a 2 | ⋯ | a n | ⎦ ⎤ .
我们将A A A i i i a i T a_i^T a i T A i , : A_{i,:} A i , :
A = [ — a 1 T — — a 2 T — ⋮ — a m T — ] . A = egin{bmatrix}
ext{ extemdash} & a_1^T & ext{ extemdash} \
ext{ extemdash} & a_2^T & ext{ extemdash} \
& vdots & \
ext{ extemdash} & a_m^T & ext{ extemdash} \
end{bmatrix}. A = ⎣ ⎡ — — — a 1 T a 2 T ⋮ a m T — — — ⎦ ⎤ .
在许多情况下，将矩阵视为列向量或行向量的集合是非常重要和方便的。一般来说，在数学上(和概念上)向量级别上的操作比标量级别上的操作会更简洁。表示矩阵的列或行没有通用约定，因此你可以使用任何符号明确定义它。
矩阵A ∈ R m × n A in Bbb{R}^{m imes n} A ∈ R m × n B ∈ R n × p B in Bbb{R}^{n imes p} B ∈ R n × p C = A B ∈ R m × p C = AB in Bbb{R}^{m imes p} C = A B ∈ R m × p C i j = ∑ k = 1 n A i k B k j C_{ij} = displaystylesum_{k=1}^n {A_{ik}B_{kj}} C ij = k = 1 ∑ n A ik B kj A A A B B B
给两个向量x , y ∈ R n oldsymbol{x},oldsymbol{y} in Bbb{R}^n x , y ∈ R n x T y oldsymbol{x}^T oldsymbol{y} x T y 内积 或者点积 ，结果是个实数：
x T y ∈ R = [ x 1 x 2 ⋯ x n ] [ y 1 y 2 ⋮ y n ] = ∑ i = 1 n x i y i oldsymbol{x}^T oldsymbol{y} in Bbb{R} =
egin{bmatrix}x_1 & x_2 & cdots & x_nend{bmatrix}
egin{bmatrix}y_1 \ y_2 \ vdots \ y_n end{bmatrix}
= sum_{i=1}^n{x_iy_i} x T y ∈ R = [ x 1 x 2 ⋯ x n ] ⎣ ⎡ y 1 y 2 ⋮ y n ⎦ ⎤ = i = 1 ∑ n x i y i
请注意，内积实际上只是矩阵乘法的特例。x T y = y T x oldsymbol{x}^T oldsymbol{y} = oldsymbol{y}^T oldsymbol{x} x T y = y T x x ∈ R m , y ∈ R n oldsymbol{x} in Bbb{R}^m , oldsymbol{y} in Bbb{R}^n x ∈ R m , y ∈ R n m m m n n n x y T ∈ R m × n oldsymbol{x} oldsymbol{y}^T in Bbb{R}^{m imes n} x y T ∈ R m × n 外积 ，它是一个矩阵，由( x y T ) i j = x i y j (oldsymbol{x} oldsymbol{y}^T)_{ij} = x_iy_j ( x y T ) ij = x i y j
x y T ∈ R m × n = [ x 1 x 2 ⋮ x m ] [ y 1 y 2 ⋯ y n ] = [ x 1 y 1 x 1 y 2 ⋯ x 1 y n x 2 y 1 x 2 y 2 ⋯ x 2 y n ⋮ ⋮ ⋱ ⋮ x m y 1 x m y 2 ⋯ x m y n ] oldsymbol{x} oldsymbol{y}^T in Bbb{R}^{m imes n}=
egin{bmatrix}x_1 \ x_2 \ vdots \ x_m end{bmatrix}
egin{bmatrix}y_1 & y_2 & cdots & y_nend{bmatrix}=
egin{bmatrix}
x_1y_1 & x_1y_2 & cdots & x_1y_n \
x_2y_1 & x_2y_2 & cdots & x_2y_n \
vdots & vdots & ddots & vdots \
x_my_1 & x_my_2 & cdots & x_my_n
end{bmatrix} x y T ∈ R m × n = ⎣ ⎡ x 1 x 2 ⋮ x m ⎦ ⎤ [ y 1 y 2 ⋯ y n ] = ⎣ ⎡ x 1 y 1 x 2 y 1 ⋮ x m y 1 x 1 y 2 x 2 y 2 ⋮ x m y 2 ⋯ ⋯ ⋱ ⋯ x 1 y n x 2 y n ⋮ x m y n ⎦ ⎤
举一个外积如何使用的一个例子：让1 ∈ R n oldsymbol{1}in Bbb{R}^{n} 1 ∈ R n n n n A ∈ R m × n A in Bbb{R}^{m imes n} A ∈ R m × n x ∈ R m oldsymbol{x} in R^{m} x ∈ R m A A A
A = [ | | | x x ⋯ x | | | ] = [ x 1 x 1 ⋯ x 1 x 2 x 2 ⋯ x 2 ⋮ ⋮ ⋱ ⋮ x m x m ⋯ x m ] = [ x 1 x 2 ⋮ x m ] [ 1 1 ⋯ 1 ] = x 1 T A=egin{bmatrix}
ext{ extbar} & ext{ extbar} & & ext{ extbar} \
x & x & cdots & x \
ext{ extbar} & ext{ extbar} & & ext{ extbar}
end{bmatrix}=
egin{bmatrix}
x_{1} & x_{1} & cdots & x_{1} \
x_{2} & x_{2} & cdots & x_{2} \
vdots & vdots & ddots & vdots \
x_{m} & x_{m} & cdots & x_{m}
end{bmatrix}=
egin{bmatrix}x_1 \ x_2 \ vdots \ x_m end{bmatrix}
egin{bmatrix}1 & 1 & cdots & 1end{bmatrix}=oldsymbol{x}oldsymbol{1}^T A = ⎣ ⎡ | x | | x | ⋯ | x | ⎦ ⎤ = ⎣ ⎡ x 1 x 2 ⋮ x m x 1 x 2 ⋮ x m ⋯ ⋯ ⋱ ⋯ x 1 x 2 ⋮ x m ⎦ ⎤ = ⎣ ⎡ x 1 x 2 ⋮ x m ⎦ ⎤ [ 1 1 ⋯ 1 ] = x 1 T
给定矩阵 A ∈ R m × n A in mathbb{R}^{m imes n} A ∈ R m × n x ∈ R n oldsymbol{x} in mathbb{R}^{n} x ∈ R n y = A x ∈ R m oldsymbol{y} = Aoldsymbol{x} in mathbb{R}^{m} y = A x ∈ R m
如果我们按行写A A A A x Aoldsymbol{x} A x
y = A x = [ — a 1 T — — a 2 T — ⋮ — a m T — ] x = [ a 1 T x a 2 T x ⋮ a m T x ] oldsymbol{y} = Aoldsymbol{x} =
egin{bmatrix}
ext{ extemdash} & a_1^T & ext{ extemdash} \
ext{ extemdash} & a_2^T & ext{ extemdash} \
& vdots & \
ext{ extemdash} & a_m^T & ext{ extemdash} \
end{bmatrix}oldsymbol{x}=
egin{bmatrix} a_1^Toldsymbol{x} \ a_2^Toldsymbol{x} \ vdots \ a_m^Toldsymbol{x} end{bmatrix} y = A x = ⎣ ⎡ — — — a 1 T a 2 T ⋮ a m T — — — ⎦ ⎤ x = ⎣ ⎡ a 1 T x a 2 T x ⋮ a m T x ⎦ ⎤
换句话说，第i i i y y y A A A i i i x oldsymbol{x} x y i = a i T x y_i=a_{i}^{T} oldsymbol{x} y i = a i T x
同样的， 可以把 A A A
y = A x = [ | | | a 1 a 2 ⋯ a n | | | ] [ x 1 x 2 ⋮ x n ] = [ a 1 ] x 1 + [ a 2 ] x 2 + ⋯ + [ a n ] x n (1) oldsymbol{y} = Aoldsymbol{x} =
egin{bmatrix}
ext{ extbar} & ext{ extbar} & & ext{ extbar} \
a^1 & a^2 & cdots & a^n \
ext{ extbar} & ext{ extbar} & & ext{ extbar}
end{bmatrix}
egin{bmatrix}x_1 \ x_2 \ vdots \ x_n end{bmatrix}=
[a^1]x_1 + [a^2]x_2 + cdots +[a^n]x_n
label{1} ag{1} y = A x = ⎣ ⎡ | a 1 | | a 2 | ⋯ | a n | ⎦ ⎤ ⎣ ⎡ x 1 x 2 ⋮ x n ⎦ ⎤ = [ a 1 ] x 1 + [ a 2 ] x 2 + ⋯ + [ a n ] x n ( 1 )
换句话说，y oldsymbol{y} y A A A 线性组合 ，其中线性组合的系数由x oldsymbol{x} x
到目前为止，我们一直是矩阵右乘一个列向量，但也可以是矩阵左乘一个行向量。如这样表示：y T = x T A oldsymbol{y}^T = oldsymbol{x}^TA y T = x T A A ∈ R m × n A in mathbb{R}^{m imes n} A ∈ R m × n x ∈ R m oldsymbol{x} in mathbb{R}^{m} x ∈ R m y ∈ R n oldsymbol{y} in mathbb{R}^{n} y ∈ R n y T oldsymbol{y}^T y T A A A
首先，我们把A A A
y T = x T A = x T [ | | | a 1 a 2 ⋯ a n | | | ] = [ x T a 1 x T a 2 ⋯ x T a n ] oldsymbol{y}^T = oldsymbol{x}^TA = oldsymbol{x}^T
egin{bmatrix}
ext{ extbar} & ext{ extbar} & & ext{ extbar} \
a^1 & a^2 & cdots & a^n \
ext{ extbar} & ext{ extbar} & & ext{ extbar}
end{bmatrix}=
egin{bmatrix}oldsymbol{x}^Ta^1 & oldsymbol{x}^Ta^2 & cdots & oldsymbol{x}^Ta^n end{bmatrix} y T = x T A = x T ⎣ ⎡ | a 1 | | a 2 | ⋯ | a n | ⎦ ⎤ = [ x T a 1 x T a 2 ⋯ x T a n ]
这表明y T oldsymbol{y}^T y T i i i x oldsymbol{x} x A A A i i i
最后，根据行表示A A A
y T = x T A = [ x 1 x 2 ⋯ x n ] [ — a 1 T — — a 2 T — ⋮ — a m T — ] = x 1 [ — a 1 T — ] + x 2 [ — a 2 T — ] + ⋯ + x n [ — a n T — ] egin{aligned}
oldsymbol{y}^T &= oldsymbol{x}^TA \&=
egin{bmatrix} x_1 & x_2 & cdots & x_n end{bmatrix}
egin{bmatrix}
ext{ extemdash} & a_1^T & ext{ extemdash} \
ext{ extemdash} & a_2^T & ext{ extemdash} \
& vdots & \
ext{ extemdash} & a_m^T & ext{ extemdash} \
end{bmatrix}\&=
x_1egin{bmatrix} ext{ extemdash} & a_1^T & ext{ extemdash}end{bmatrix}+
x_2egin{bmatrix} ext{ extemdash} & a_2^T & ext{ extemdash}end{bmatrix}+ cdots +
x_negin{bmatrix} ext{ extemdash} & a_n^T & ext{ extemdash}end{bmatrix}
end{aligned} y T = x T A = [ x 1 x 2 ⋯ x n ] ⎣ ⎡ — — — a 1 T a 2 T ⋮ a m T — — — ⎦ ⎤ = x 1 [ — a 1 T — ] + x 2 [ — a 2 T — ] + ⋯ + x n [ — a n T — ]
所以我们看到y T oldsymbol{y}^T y T A A A x oldsymbol{x} x
有了这些知识，我们现在可以看看四种不同的（当然是等价的）查看矩阵与矩阵乘法C = A B C = AB C = A B
首先，我们可以将矩阵-矩阵乘法视为一组向量-向量乘积。 从定义中可以得出：最明显的观点是C C C i , j i,j i , j A A A i i i B B B j j j
C = A B = [ — a 1 T — — a 2 T — ⋮ — a m T — ] [ | | | b 1 b 2 ⋯ b p | | | ] = [ a 1 b 1 a 1 b 2 ⋯ a 1 b p a 2 b 1 a 2 b 2 ⋯ a 2 b p ⋮ ⋮ ⋱ ⋮ a m b 1 a m b 2 ⋯ a m b p ] C = AB =
egin{bmatrix}
ext{ extemdash} & a_1^T & ext{ extemdash} \
ext{ extemdash} & a_2^T & ext{ extemdash} \
& vdots & \
ext{ extemdash} & a_m^T & ext{ extemdash} \
end{bmatrix}
egin{bmatrix}
ext{ extbar} & ext{ extbar} & & ext{ extbar} \
b^1 & b^2 & cdots & b^p \
ext{ extbar} & ext{ extbar} & & ext{ extbar}
end{bmatrix}=
egin{bmatrix}
a_1b_1 & a_1b_2 & cdots & a_1b_p \
a_2b_1 & a_2b_2 & cdots & a_2b_p \
vdots & vdots & ddots & vdots \
a_mb_1 & a_mb_2 & cdots & a_mb_p
end{bmatrix} C = A B = ⎣ ⎡ — — — a 1 T a 2 T ⋮ a m T — — — ⎦ ⎤ ⎣ ⎡ | b 1 | | b 2 | ⋯ | b p | ⎦ ⎤ = ⎣ ⎡ a 1 b 1 a 2 b 1 ⋮ a m b 1 a 1 b 2 a 2 b 2 ⋮ a m b 2 ⋯ ⋯ ⋱ ⋯ a 1 b p a 2 b p ⋮ a m b p ⎦ ⎤
这里的矩阵A ∈ R m × n , B ∈ R n × p A in Bbb{R}^{m imes n} , B in Bbb{R}^{n imes p} A ∈ R m × n , B ∈ R n × p a i ∈ R n , b j ∈ R n a_i in Bbb{R}^n , b^j in Bbb{R}^n a i ∈ R n , b j ∈ R n A A A B B B A A A B B B A B AB A B
C = A B = [ | | | a 1 a 2 ⋯ a n | | | ] [ — b 1 T — — b 2 T — ⋮ — b n T — ] = ∑ i = 1 n a i b i T C = AB =
egin{bmatrix}
ext{ extbar} & ext{ extbar} & & ext{ extbar} \
a^1 & a^2 & cdots & a^n \
ext{ extbar} & ext{ extbar} & & ext{ extbar}
end{bmatrix}
egin{bmatrix}
ext{ extemdash} & b_1^T & ext{ extemdash} \
ext{ extemdash} & b_2^T & ext{ extemdash} \
& vdots & \
ext{ extemdash} & b_n^T & ext{ extemdash} \
end{bmatrix}=
sum_{i=1}^n{a^ib_i^T} C = A B = ⎣ ⎡ | a 1 | | a 2 | ⋯ | a n | ⎦ ⎤ ⎣ ⎡ — — — b 1 T b 2 T ⋮ b n T — — — ⎦ ⎤ = i = 1 ∑ n a i b i T
换句话说，A B AB A B A A A i i i B B B i i i a i ∈ R m a^i in mathbb{R}^ m a i ∈ R m b i ∈ R p b_i in mathbb{R}^p b i ∈ R p a i b i T a^ib_i^T a i b i T m × p m×p m × p C C C
其次，我们还可以将矩阵-矩阵乘法视为一组矩阵-向量乘法。如果我们把B B B C C C A A A B B B
C = A B = A [ | | | b 1 b 2 ⋯ b p | | | ] = [ | | | A b 1 A b 2 ⋯ A b p | | | ] (2) C = AB = A
egin{bmatrix}
ext{ extbar} & ext{ extbar} & & ext{ extbar} \
b^1 & b^2 & cdots & b^p \
ext{ extbar} & ext{ extbar} & & ext{ extbar}
end{bmatrix}=
egin{bmatrix}
ext{ extbar} & ext{ extbar} & & ext{ extbar} \
Ab^1 & Ab^2 & cdots & Ab^p \
ext{ extbar} & ext{ extbar} & & ext{ extbar}
end{bmatrix}
label{2} ag{2} C = A B = A ⎣ ⎡ | b 1 | | b 2 | ⋯ | b p | ⎦ ⎤ = ⎣ ⎡ | A b 1 | | A b 2 | ⋯ | A b p | ⎦ ⎤ ( 2 )
这里C C C i i i c i = A b i c_i = Ab_i c i = A b i
最后，我们有类似的观点，我们用行表示A A A C C C A A A B B B
C = A B = [ — a 1 T — — a 2 T — ⋮ — a m T — ] B = [ — a 1 T B — — a 2 T B — ⋮ — a m T B — ] C = AB =
egin{bmatrix}
ext{ extemdash} & a_1^T & ext{ extemdash} \
ext{ extemdash} & a_2^T & ext{ extemdash} \
& vdots & \
ext{ extemdash} & a_m^T & ext{ extemdash} \
end{bmatrix}B=
egin{bmatrix}
ext{ extemdash} & a_1^TB & ext{ extemdash} \
ext{ extemdash} & a_2^TB & ext{ extemdash} \
& vdots & \
ext{ extemdash} & a_m^TB & ext{ extemdash} \
end{bmatrix} C = A B = ⎣ ⎡ — — — a 1 T a 2 T ⋮ a m T — — — ⎦ ⎤ B = ⎣ ⎡ — — — a 1 T B a 2 T B ⋮ a m T B — — — ⎦ ⎤
这里C C C i i i c i T = a i T B c_i^T = a_i^T B c i T = a i T B
将矩阵乘法剖析到如此大的程度似乎有点矫枉过正，特别是当所有这些观点都紧跟在我们在本节开头给出的初始定义（C = A B C=AB C = A B
这些不同方法的直接优势在于它们允许您在向量的级别/单位而不是标量上进行操作 。 为了完全理解线性代数而不会迷失在复杂的索引操作中，关键是操作尽可能大(向量而不是标量)的概念。
实际上所有的线性代数都是在处理某种矩阵乘法，多花一些时间对这里提出的观点进行直观的理解是非常必要的。
除此之外，你还应该了解一些更高级别的矩阵乘法的基本性质：
矩阵乘法结合律: ( A B ) C = A ( B C ) (AB)C = A(BC) ( A B ) C = A ( BC )
矩阵乘法分配律: A ( B + C ) = A B + A C A(B + C) = AB + AC A ( B + C ) = A B + A C
矩阵乘法一般是不可交换的; 也就是说，通常A B ≠ B A AB
e BA A B � = B A A ∈ R m × n ， A in mathbb{R}^ {m imes n}， A ∈ R m × n ， B ∈ R n × p B in mathbb{R}^ {n imes p} B ∈ R n × p m m m q q q B A BA B A
如果您不熟悉这些性质，请花点时间自己验证它们。 例如，为了检查矩阵乘法的结合性，假设A ∈ R m × n ， A in mathbb{R}^ {m imes n}， A ∈ R m × n ， B ∈ R n × p B in mathbb{R}^ {n imes p} B ∈ R n × p C ∈ R p × q C in mathbb{R}^ {p imes q} C ∈ R p × q A B ∈ R m × p AB in mathbb{R}^ {m imes p} A B ∈ R m × p ( A B ) C ∈ R m × q (AB)C in mathbb{R}^ {m imes q} ( A B ) C ∈ R m × q B C ∈ R n × q BC in mathbb{R}^ {n imes q} BC ∈ R n × q A ( B C ) ∈ R m × q A(BC) in mathbb{R}^ {m imes q} A ( BC ) ∈ R m × q ( A B ) C (AB)C ( A B ) C i , j i,j i , j A ( B C ) A(BC) A ( BC ) i , j i,j i , j
( ( A B ) C ) i j = ∑ k = 1 p ( A B ) i k C k j = ∑ k = 1 p ( ∑ l = 1 n A i l B l k ) C k j = ∑ k = 1 p ( ∑ l = 1 n A i l B l k C k j ) = ∑ l = 1 n ( ∑ k = 1 p A i l B l k C k j ) = ∑ l = 1 n A i l ( ∑ k = 1 p B l k C k j ) = ∑ l = 1 n A i l ( B C ) l j = ( A ( B C ) ) i j egin{aligned} % aligned &= 等号对齐
((A B) C)_{ij} &= sum_{k=1}^p{(AB)_{ik}C_{kj}} = sum_{k=1}^p Bigg( sum_{l=1}^n{A_{il}B_{lk}} Bigg) C_{kj} \
&=sum_{k=1}^p Bigg( sum_{l=1}^n{A_{il}B_{lk}C_{kj}}Bigg) = sum_{l=1}^n Bigg( sum_{k=1}^p{A_{il}B_{lk}C_{kj}}Bigg)\
&=sum_{l=1}^nA_{il}Bigg(sum_{k=1}^p{B_{lk}C_{kj}}Bigg) = sum_{l=1}^n{A_{il}(BC)_{lj}} = (A(BC))_{ij}
end{aligned} (( A B ) C ) ij = k = 1 ∑ p ( A B ) ik C kj = k = 1 ∑ p ( l = 1 ∑ n A i l B l k ) C kj = k = 1 ∑ p ( l = 1 ∑ n A i l B l k C kj ) = l = 1 ∑ n ( k = 1 ∑ p A i l B l k C kj ) = l = 1 ∑ n A i l ( k = 1 ∑ p B l k C kj ) = l = 1 ∑ n A i l ( BC ) l j = ( A ( BC ) ) ij
这里，第一个和最后两个等式简单地使用了矩阵乘法的定义，第三个和第五个等式使用了标量乘法对加法的分配性质，第四个等式使用了标量加法的交换性和结合性。 这种通过简化为简单标量性质来证明矩阵性质的技术会经常出现，因此请确保您熟悉它。
在本节中，我们将介绍矩阵和向量的几种操作和性质。 希望其中的大部分内容都可以帮您复习，此笔记可以作为参考。
单位矩阵 I ∈ R n × n I in Bbb{R}^{n imes n} I ∈ R n × n
I i j = { 1 i = j 0 i ≠ j . I_{ij} =
egin{cases}
1 & i=j \
0 & i
eq j
end{cases}. I ij = { 1 0 i = j i � = j .
对于所有矩阵A ∈ R m × n A in mathbb{R}^ {m imes n} A ∈ R m × n A I = A = I A AI=A=IA A I = A = I A I I I I I I A I = A AI = A A I = A I I I n × n n imes n n × n A = I A A = IA A = I A I I I m × m m imes m m × m
对角矩阵 D = d i a g ( d 1 , d 2 , ⋯ , d n ) D=diag(d_1, d_2,cdots, d_n) D = d ia g ( d 1 , d 2 , ⋯ , d n )
D i j = { d i i = j 0 i ≠ j . D_{ij} =
egin{cases}
d_i & i=j \
0 & i
eq j
end{cases}. D ij = { d i 0 i = j i � = j .
很明显，单位矩阵I = d i a g ( 1 , 1 , ⋯ , 1 ) I= diag(1, 1, cdots , 1) I = d ia g ( 1 , 1 , ⋯ , 1 )
矩阵的转置 A ∈ R m × n A in Bbb{R}^{m imes n} A ∈ R m × n A T ∈ R n × m A^T in Bbb{R}^{n imes m} A T ∈ R n × m
( A T ) i j = A j i . (A^T)_{ij} = A_{ji}. ( A T ) ij = A ji .
事实上，我们在描述行向量时已经使用了转置，因为列向量的转置自然是行向量。
转置有以下性质，且很容易验证：
( A T ) T = A (A^T)^T = A ( A T ) T = A ( A B ) T = B T A T (AB)^T = B^TA^T ( A B ) T = B T A T ( A + B ) T = A T + B T (A+B)^T = A^T + B^T ( A + B ) T = A T + B T
如果A = A T A = A^T A = A T 方阵 A ∈ R n × n A in Bbb{R}^{n imes n} A ∈ R n × n 对称
元素满足a i j = a j i , ∀ i , j a_{ij} = a_{ji} , forall i,j a ij = a ji , ∀ i , j
A = A T A = A^T A = A T 对于任意方阵A A A A + A T A + A^T A + A T
对角矩阵都是对称矩阵
如果A = − A T A = -A^T A = − A T 反对称
元素满足a i j = − a j i , ∀ i , j a_{ij} = -a_{ji} , forall i,j a ij = − a ji , ∀ i , j i = j i=j i = j a i j = 0 a_{ij} = 0 a ij = 0
A , B A,B A , B A ± B A plusmn B A ± B 若A A A B B B A B − B A AB-BA A B − B A
很容易证明，对于任何矩阵A ∈ R n × n A in mathbb{R}^ {n imes n} A ∈ R n × n A + A T A + A^ T A + A T A − A T A -A^T A − A T
由此得出，任意方矩阵A ∈ R n × n A in mathbb{R}^ {n imes n} A ∈ R n × n
A = 1 2 ( A + A T ) + 1 2 ( A − A T ) A=frac{1}{2}(A+A^T)+frac{1}{2}(A-A^T) A = 2 1 ( A + A T ) + 2 1 ( A − A T )
事实证明，对称矩阵在实践中用到很多，它们有很多很好的性质，我们很快就会看到它们。n n n S n mathbb{S}^n S n A ∈ S n A in mathbb{S}^n A ∈ S n A A A n × n n imes n n × n
方矩阵A ∈ R n × n A in mathbb{R}^ {n imes n} A ∈ R n × n 迹 tr ( A ) operatorname{tr} (A) tr ( A ) tr A operatorname{tr} A tr A
tr A = ∑ i = 1 n A i i operatorname{tr} A=sum_{i=1}^{n} A_{i i} tr A = i = 1 ∑ n A ii
如 CS229 讲义中所述，迹具有以下性质（如下所示）：
对于矩阵A ∈ R n × n A in mathbb{R}^ {n imes n} A ∈ R n × n tr A = tr A T operatorname{tr}A =operatorname{tr}A^T tr A = tr A T
对于矩阵A , B ∈ R n × n A,B in mathbb{R}^ {n imes n} A , B ∈ R n × n tr ( A + B ) = tr A + tr B operatorname{tr}(A + B) = operatorname{tr}A + operatorname{tr}B tr ( A + B ) = tr A + tr B
对于矩阵A ∈ R n × n A in mathbb{R}^ {n imes n} A ∈ R n × n t ∈ R t in mathbb{R} t ∈ R tr ( t A ) = t tr A operatorname{tr}(tA) = toperatorname{tr}A tr ( t A ) = t tr A
对于矩阵 A A A B B B A B AB A B tr A B = tr B A operatorname{tr}AB = operatorname{tr}BA tr A B = tr B A
对于矩阵 A A A B B B C C C A B C ABC A BC tr A B C = tr B C A = tr C A B operatorname{tr}ABC = operatorname{tr}BCA=operatorname{tr}CAB tr A BC = tr BC A = tr C A B
我们给出第四个性质的证明。假设A ∈ R m × n A in mathbb{R}^ {m imes n} A ∈ R m × n B ∈ R n × m B in mathbb{R}^ {n imes m} B ∈ R n × m A B ∈ R m × m AB in mathbb{R}^ {m imes m} A B ∈ R m × m B A ∈ R n × n BA in mathbb{R}^ {n imes n} B A ∈ R n × n tr A B = tr B A operatorname{tr}AB = operatorname{tr}BA tr A B = tr B A
tr A B = ∑ i = 1 m ( A B ) i i = ∑ i = 1 m ( ∑ j = 1 n A i j B j i ) = ∑ i = 1 m ∑ j = 1 n A i j B j i = ∑ j = 1 n ∑ i = 1 m B j i A i j = ∑ j = 1 n ( ∑ i = 1 m B j i A i j ) = ∑ j = 1 n ( B A ) j j = tr B A egin{aligned}
operatorname{tr} A B &=sum_{i=1}^{m}(A B)_{i i}=sum_{i=1}^{m}left(sum_{j=1}^{n} A_{i j} B_{j i}
ight) \
&=sum_{i=1}^{m} sum_{j=1}^{n} A_{i j} B_{j i}=sum_{j=1}^{n} sum_{i=1}^{m} B_{j i} A_{i j} \
&=sum_{j=1}^{n}left(sum_{i=1}^{m} B_{j i} A_{i j}
ight)=sum_{j=1}^{n}(B A)_{j j}=operatorname{tr} B A
end{aligned} tr A B = i = 1 ∑ m ( A B ) ii = i = 1 ∑ m ( j = 1 ∑ n A ij B ji ) = i = 1 ∑ m j = 1 ∑ n A ij B ji = j = 1 ∑ n i = 1 ∑ m B ji A ij = j = 1 ∑ n ( i = 1 ∑ m B ji A ij ) = j = 1 ∑ n ( B A ) jj = tr B A
这里，第一个和最后两个等式使用了迹运算符和矩阵乘法的定义。 重点在第四个等式,使用标量乘法的交换性来反转每个乘积中的项的顺序，以及标量加法的交换性和结合性来重新排列求和的顺序。
向量的范数 ∥ x ∥ |x| ∥ x ∥ ℓ 2 ell_{2} ℓ 2
∥ x ∥ 2 = ∑ i = 1 n x i 2 |x|_{2}=sqrt{sum_{i=1}^{n} x_{i}^{2}} ∥ x ∥ 2 = i = 1 ∑ n x i 2
注意：∥ x ∥ 2 2 = x T x |x|_{2}^{2}=x^{T} x ∥ x ∥ 2 2 = x T x
更正式地，范数是满足 4 个性质的函数（f : R n → R f : mathbb{R}^{n}
ightarrow mathbb{R} f : R n → R
对于所有的 x ∈ R n x in mathbb{R}^ {n} x ∈ R n f ( x ) ≥ 0 f(x) geq 0 f ( x ) ≥ 0
当且仅当x = 0 x = 0 x = 0 f ( x ) = 0 f(x) = 0 f ( x ) = 0
对于所有x ∈ R n x in mathbb{R}^ {n} x ∈ R n t ∈ R tin mathbb{R} t ∈ R f ( t x ) = ∣ t ∣ f ( x ) f(tx) = left| t
ight|f(x) f ( t x ) = ∣ t ∣ f ( x )
对于所有 x , y ∈ R n x,y in mathbb{R}^ {n} x , y ∈ R n f ( x + y ) ≤ f ( x ) + f ( y ) f(x + y) leq f(x) + f(y) f ( x + y ) ≤ f ( x ) + f ( y )
其他范数的例子，如：ℓ 1 ell_1 ℓ 1
∥ x ∥ 1 = ∑ i = 1 n ∣ x i ∣ |x|_{1}=sum_{i=1}^{n}|x_{i}| ∥ x ∥ 1 = i = 1 ∑ n ∣ x i ∣
和ℓ ∞ ell_{infty } ℓ ∞
∥ x ∥ ∞ = max i ∣ x i ∣ |x|_{infty}=max_{i}left|x_{i}
ight| ∥ x ∥ ∞ = i max ∣ x i ∣
事实上，到目前为止所提出的所有三个范数都是ℓ p ell_p ℓ p p ≥ 1 p geq 1 p ≥ 1
∥ x ∥ p = ( ∑ i = 1 n ∣ x i ∣ p ) 1 / p |x|_{p}=left(sum_{i=1}^{n}left|x_{i}
ight|^{p}
ight)^{1 / p} ∥ x ∥ p = ( i = 1 ∑ n ∣ x i ∣ p ) 1/ p
也可以为矩阵定义范数，例如Frobenius 范数:
∥ A ∥ F = ∑ i = 1 m ∑ j = 1 n A i j 2 = tr ( A T A ) |A|_{F}=sqrt{sum_{i=1}^{m} sum_{j=1}^{n} A_{i j}^{2}}=sqrt{operatorname{tr}left(A^{T} A
ight)} ∥ A ∥ F = i = 1 ∑ m j = 1 ∑ n A ij 2 = tr ( A T A )
还有很多其他范数，但它们超出了这个复习材料的范围。
一个向量集合{ x 1 , x 2 , ⋯ x n } ⊂ R m { x_1,x_2, cdots x_n } subset mathbb{R}^m { x 1 , x 2 , ⋯ x n } ⊂ R m 线性无关 线性相关
x j = ∑ i = 1 , i ≠ j n α i x i x_{j}=sum_{i=1,i
eq j}^{n} alpha_{i} x_{i} x j = i = 1 , i � = j ∑ n α i x i
存在α 1 , ⋯ α n ∈ R alpha_1,cdots alpha_{n} in mathbb{R} α 1 , ⋯ α n ∈ R x 1 , x 2 , ⋯ x n x_1,x_2, cdots x_n x 1 , x 2 , ⋯ x n 线性相关 的描述（存在不全为零的数α i alpha_{i} α i
∑ i = 1 n α i x i = 0 , ∃ α i ≠ 0 sum_{i=1}^{n} alpha_{i} x_{i} = 0,exists alpha_i
eq 0 i = 1 ∑ n α i x i = 0 , ∃ α i � = 0
例如，向量：
x 1 = [ 1 2 3 ] x 2 = [ 4 1 5 ] x 3 = [ 2 − 3 − 1 ] x_{1}=
egin{bmatrix}
1 \
2 \
3
end{bmatrix} quad
x_{2}=
egin{bmatrix}
4 \
1 \
5
end{bmatrix} quad
x_{3}=
egin{bmatrix}
2 \
-3 \
-1
end{bmatrix} x 1 = ⎣ ⎡ 1 2 3 ⎦ ⎤ x 2 = ⎣ ⎡ 4 1 5 ⎦ ⎤ x 3 = ⎣ ⎡ 2 − 3 − 1 ⎦ ⎤
是线性相关的，因为：x 3 = − 2 x 1 + x 2 x_3=-2x_1+x_2 x 3 = − 2 x 1 + x 2
矩阵A ∈ R m × n A in mathbb{R}^{m imes n} A ∈ R m × n 列秩 A A A A A A 行秩 A A A A ∈ R m × n A in mathbb{R}^{m imes n} A ∈ R m × n A A A A A A A A A 秩 rank ( A ) ext{rank}(A) rank ( A )
对于 A ∈ R m × n A in mathbb{R}^{m imes n} A ∈ R m × n rank ( A ) ≤ m i n ( m , n ) ext{rank}(A) leq min(m, n) rank ( A ) ≤ min ( m , n ) ( A ) = min ( m , n ) ext(A) = ext{min} (m, n) ( A ) = min ( m , n ) A A A 满秩
对于 A ∈ R m × n A in mathbb{R}^{m imes n} A ∈ R m × n rank ( A ) = rank ( A T ) ext{rank}(A) = ext{rank}(A^T) rank ( A ) = rank ( A T )
对于 A ∈ R m × n A in mathbb{R}^{m imes n} A ∈ R m × n B ∈ R n × p B in mathbb{R}^{n imes p} B ∈ R n × p rank ( A B ) ≤ min ( rank ( A ) , rank ( B ) ) ext{rank}(AB) leq ext{min} ( ext{rank}(A), ext{rank}(B)) rank ( A B ) ≤ min ( rank ( A ) , rank ( B ))
对于 A , B ∈ R m × n A,B in mathbb{R}^{m imes n} A , B ∈ R m × n rank ( A + B ) ≤ rank ( A ) + rank ( B ) ext{rank}(A + B) leq ext{rank}(A) + ext{rank}(B) rank ( A + B ) ≤ rank ( A ) + rank ( B )
方阵A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n 逆 A − 1 A^{-1} A − 1
A − 1 A = I = A A − 1 A^{-1}A=I=AA^{-1} A − 1 A = I = A A − 1
请注意，并非所有矩阵都具有逆。 例如，非方形矩阵根据定义没有逆(存在伪逆 )。 然而，对于一些方形矩阵A A A A − 1 A^{-1} A − 1 A − 1 A^{-1} A − 1 A A A 可逆 非奇异 不可逆 奇异
为了使方阵 A 具有逆A − 1 A^{-1} A − 1 A A A A , B ∈ R n × n A,B in mathbb{R}^{n imes n} A , B ∈ R n × n
( A − 1 ) − 1 = A (A^{-1})^{-1} = A ( A − 1 ) − 1 = A ( A B ) − 1 = B − 1 A − 1 (AB)^{-1} = B^{-1}A^{-1} ( A B ) − 1 = B − 1 A − 1 ( A − 1 ) T = ( A T ) − 1 (A^{-1})^{T} =(A^{T})^{-1} ( A − 1 ) T = ( A T ) − 1 A − T A^{-T} A − T
作为如何使用逆的示例，考虑线性方程组，A x = b Ax = b A x = b A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n x , b ∈ R x,bin mathbb{R} x , b ∈ R A A A x = A − 1 b x = A^{-1}b x = A − 1 b A ∈ R m × n A in mathbb{R}^{m imes n} A ∈ R m × n 伪逆 ）
如果 x T y = 0 x^Ty=0 x T y = 0 x , y ∈ R n x,yin mathbb{R}^{n} x , y ∈ R n 正交 ∥ x ∥ 2 = 1 |x|_2=1 ∥ x ∥ 2 = 1 x ∈ R n xin mathbb{R}^{n} x ∈ R n 归一化 U ∈ R n × n Uin mathbb{R}^{n imes n} U ∈ R n × n U U U 正交矩阵 （注意在讨论向量与矩阵时的意义不一样，两个向量正交只需要内积为 0，正交矩阵是各列相互正交并且被归一化 ）。
它可以从正交性和正态性的定义中得出:
U T U = I = U U T U^ TU = I = U U^T U T U = I = U U T
换句话说，正交矩阵的逆是其转置。 注意，如果U U U U ∈ R m × n , n < m Uin mathbb{R}^{m imes n}, n < m U ∈ R m × n , n < m U T U = I U^TU = I U T U = I U U T ≠ I UU^T
eq I U U T � = I 正交矩阵一定是方阵 。
正交矩阵的另一个好的特性是在具有正交矩阵的向量上操作不会改变其欧几里德范数，即(i.e.):
∥ U x ∥ 2 = ∥ x ∥ 2 (3) |U x|_{2}=|x|_{2}
label{3} ag{3} ∥ Ux ∥ 2 = ∥ x ∥ 2 ( 3 )
对于任何 x ∈ R n xin mathbb{R}^n x ∈ R n U ∈ R n × n Uin mathbb{R}^{n imes n} U ∈ R n × n
张成 { x 1 , x 2 , ⋯ x n } { x_1,x_2, cdots x_n } { x 1 , x 2 , ⋯ x n } { x 1 , ⋯ x n } { x_1, cdots x_n } { x 1 , ⋯ x n }
span ( { x 1 , ⋯ x n } ) = { v : v = ∑ i = 1 n α i x i , α i ∈ R } operatorname{span}({x_1, cdots x_n }) = Bigg{v:v=sum_{i=1}^n{alpha_i x_i}, alpha_i in Bbb{R} Bigg} span ({ x 1 , ⋯ x n }) = { v : v = i = 1 ∑ n α i x i , α i ∈ R }
可以看到，如果{ x 1 , ⋯ x n } {x_{1}, cdots x_{n}} { x 1 , ⋯ x n } n n n x i ∈ R n x_i in mathbb{R}^{n} x i ∈ R n span ( { x 1 , … x n } ) = R n ext{span}({x_{1}, ldots x_{n}})=mathbb{R}^{n} span ({ x 1 , … x n }) = R n v ∈ R n vin mathbb{R}^{n} v ∈ R n x 1 x_1 x 1 x n x_n x n y ∈ R m yin mathbb{R}^{m} y ∈ R m 投影 { x 1 , … x n } {x_{1}, ldots x_{n}} { x 1 , … x n } x i ∈ R m x_i in mathbb{R}^{m} x i ∈ R m v ∈ span ( { x 1 , … , x n } ) v in operatorname{span}({x_{1}, ldots, x_{n}}) v ∈ span ({ x 1 , … , x n }) ∥ v − y ∥ 2 |v - y|_2 ∥ v − y ∥ 2 v v v y y y
我们将投影表示为Proj ( y ; { x 1 , … x n } ) operatorname{Proj}left(y ;left{x_{1}, ldots x_{n}
ight}
ight) Proj ( y ; { x 1 , … x n } )
Proj ( y ; { x 1 , … x n } ) = argmin v ∈ span ( { x 1 , … , x n } ) ∥ y − v ∥ 2 operatorname{Proj}left(y ;left{x_{1}, ldots x_{n}
ight}
ight)=operatorname{argmin}_{v in operatorname{span}left(left{x_{1}, ldots, x_{n}
ight}
ight)}|y-v|_{2} Proj ( y ; { x 1 , … x n } ) = argmin v ∈ span ( { x 1 , … , x n } ) ∥ y − v ∥ 2
矩阵A ∈ R m × n Ain mathbb{R}^{m imes n} A ∈ R m × n 值域 列空间 R ( A ) mathcal{R}(A) R ( A ) A A A
R ( A ) = { v ∈ R m : v = A x , x ∈ R n } mathcal{R}(A)=left{v in mathbb{R}^{m} : v=A x, x in mathbb{R}^{n}
ight} R ( A ) = { v ∈ R m : v = A x , x ∈ R n }
做一些技术性的假设（即A A A n < m n <m n < m y ∈ R m y in mathbb{R}^{m} y ∈ R m A A A
Proj ( y ; A ) = argmin v ∈ R ( A ) ∥ v − y ∥ 2 = A ( A T A ) − 1 A T y operatorname{Proj}(y ; A)=operatorname{argmin}_{v in mathcal{R}(A)}|v-y|_{2}=Aleft(A^{T} A
ight)^{-1} A^{T} y Proj ( y ; A ) = argmin v ∈ R ( A ) ∥ v − y ∥ 2 = A ( A T A ) − 1 A T y
这个最后的方程应该看起来非常熟悉，因为它几乎与我们在课程中（我们将很快再次得出）得到的公式：与参数的最小二乘估计一样。
当A A A a ∈ R m a in mathbb{R}^{m} a ∈ R m
Proj ( y ; a ) = a a T a T a y operatorname{Proj}(y ; a)=frac{a a^{T}}{a^{T} a} y Proj ( y ; a ) = a T a a a T y
一个矩阵A ∈ R m × n Ain mathbb{R}^{m imes n} A ∈ R m × n 零空间 N ( A ) mathcal{N}(A) N ( A ) A A A
N ( A ) = { x ∈ R n : A x = 0 } mathcal{N}(A)=left{x in mathbb{R}^{n} : A x=0
ight} N ( A ) = { x ∈ R n : A x = 0 }
注意，R ( A ) mathcal{R}(A) R ( A ) m m m N ( A ) mathcal{N}(A) N ( A ) n n n R ( A T ) mathcal{R}(A^T) R ( A T ) N ( A ) mathcal{N}(A) N ( A ) R n mathbb{R}^{n} R n
{ w : w = u + v , u ∈ R ( A T ) , v ∈ N ( A ) } = R n and R ( A T ) ∩ N ( A ) = { 0 } left{w : w=u+v, u in mathcal{R}left(A^{T}
ight), v in mathcal{N}(A)
ight}=mathbb{R}^{n} ext { and } mathcal{R}left(A^{T}
ight) cap mathcal{N}(A)={mathbf{0}} { w : w = u + v , u ∈ R ( A T ) , v ∈ N ( A ) } = R n and R ( A T ) ∩ N ( A ) = { 0 }
换句话说，R ( A T ) mathcal{R}(A^T) R ( A T ) N ( A ) mathcal{N}(A) N ( A ) R n mathbb{R}^{n} R n 正交补 R ( A T ) = N ( A ) ⊥ mathcal{R}(A^T)= mathcal{N}(A)^{perp} R ( A T ) = N ( A ) ⊥
一个方阵A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n 行列式 det ext {det} det R n × n → R n mathbb{R}^{n imes n}
ightarrow mathbb{R}^{n} R n × n → R n ∣ A ∣ left| A
ight| ∣ A ∣ det A ext{det} A det A
给定一个矩阵：
[ — a 1 T — — a 2 T — ⋮ — a n T — ] egin{bmatrix}
ext{ extemdash} & a_1^T & ext{ extemdash} \
ext{ extemdash} & a_2^T & ext{ extemdash} \
& vdots & \
ext{ extemdash} & a_n^T & ext{ extemdash} \
end{bmatrix} ⎣ ⎡ — — — a 1 T a 2 T ⋮ a n T — — — ⎦ ⎤
考虑通过采用A A A a 1 , … a n ∈ R n a_{1}, ldots a_{n}in mathbb{R}^{n} a 1 , … a n ∈ R n S ⊂ R n S subset mathbb{R}^{n} S ⊂ R n S S S span ( { a 1 , … a n } ) ext{span}({a_{1}, ldots a_{n}}) span ({ a 1 , … a n }) α 1 , … α n alpha_{1}, ldots alpha_{n} α 1 , … α n α 1 , ⋯ , α n alpha_1, cdots ,alpha_n α 1 , ⋯ , α n 0 ≤ α i ≤ 1 , i = 1 , … , n 0 leq alpha_{i} leq 1, i=1, ldots, n 0 ≤ α i ≤ 1 , i = 1 , … , n
S = { v ∈ R n : v = ∑ i = 1 n α i a i where 0 ≤ α i ≤ 1 , i = 1 , … , n } S=left{v in mathbb{R}^{n} : v=sum_{i=1}^{n} alpha_{i} a_{i} ext { where } 0 leq alpha_{i} leq 1, i=1, ldots, n
ight} S = { v ∈ R n : v = i = 1 ∑ n α i a i where 0 ≤ α i ≤ 1 , i = 1 , … , n }
事实证明，A A A S S S
比方说：一个2 × 2 2 imes2 2 × 2
A = [ 1 3 3 2 ] (4) A=
egin{bmatrix}
1 & 3 \
3 & 2
end{bmatrix}
label{4} ag{4} A = [ 1 3 3 2 ] ( 4 )
它的矩阵的行是：
a 1 = [ 1 3 ] a 2 = [ 3 2 ] a_{1}=left[egin{array}{l}{1} \ {3}end{array}
ight]
quad
a_{2}=left[egin{array}{l}{3} \ {2}end{array}
ight] a 1 = [ 1 3 ] a 2 = [ 3 2 ]
对应于这些行对应的集合S S S S S S ∣ A ∣ = − 7 left| A
ight| = -7 ∣ A ∣ = − 7
在三维中，集合S S S S S S 3 × 3 3×3 3 × 3 S S S n n n
(0, 0) a1 (1, 3) a2 (3, 2) (4, 5) Viewer does not support full SVG 1.1
图 1：（4）中给出的2 × 2 2×2 2 × 2 A A A a 1 a_1 a 1 a 2 a_2 a 2 A A A S S S ∣ det A ∣ = 7 left| ext{det} A
ight| = 7 ∣ det A ∣ = 7
在代数上，行列式满足以下三个性质（所有其他性质都遵循这些性质，包括通用公式）：
单位阵的行列式为 1, ∣ I ∣ = 1 left| I
ight|= 1 ∣ I ∣ = 1
给定一个矩阵 A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n A A A t ∈ R t in mathbb{R} t ∈ R t ∣ A ∣ tleft| A
ight| t ∣ A ∣
∣ [ — t a 1 T — — a 2 T — ⋮ — a m T — ] ∣ = t ∣ A ∣ left|egin{bmatrix}
ext{ extemdash} & t a_1^T & ext{ extemdash} \
ext{ extemdash} & a_2^T & ext{ extemdash} \
& vdots & \
ext{ extemdash} & a_m^T & ext{ extemdash} \
end{bmatrix}
ight| = t|A| ∣ ∣ ⎣ ⎡ — — — t a 1 T a 2 T ⋮ a m T — — — ⎦ ⎤ ∣ ∣ = t ∣ A ∣
几何上，将集合S S S t t t t t t
如果我们交换任意两行在a i T a_i^T a i T a j T a_j^T a j T − ∣ A ∣ -left| A
ight| − ∣ A ∣ ∣ [ — a 2 T — — a 1 T — ⋮ — a m T — ] ∣ = − ∣ A ∣ left|egin{bmatrix}
ext{ extemdash} & a_2^T & ext{ extemdash} \
ext{ extemdash} & a_1^T & ext{ extemdash} \
& vdots & \
ext{ extemdash} & a_m^T & ext{ extemdash} \
end{bmatrix}
ight| = -|A| ∣ ∣ ⎣ ⎡ — — — a 2 T a 1 T ⋮ a m T — — — ⎦ ⎤ ∣ ∣ = − ∣ A ∣
从上述三个性质中得出的几个性质包括：
对于 A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n ∣ A ∣ = ∣ A T ∣ left| A
ight| = left| A^T
ight| ∣ A ∣ = ∣ ∣ A T ∣ ∣
对于 A , B ∈ R n × n A,B in mathbb{R}^{n imes n} A , B ∈ R n × n ∣ A B ∣ = ∣ A ∣ ∣ B ∣ left| AB
ight|= left| A
ight|left| B
ight| ∣ A B ∣ = ∣ A ∣ ∣ B ∣
对于 A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n ∣ A ∣ = 0 left| A
ight|= 0 ∣ A ∣ = 0 A A A A A A S S S n n n
对于 A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n A A A ∣ A − 1 ∣ = 1 / ∣ A ∣ left| A ^{−1}
ight| = 1/left| A
ight| ∣ ∣ A −1 ∣ ∣ = 1/ ∣ A ∣
在给出行列式的一般定义之前，我们定义，对于A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n A i , j ∈ R ( n − 1 ) × ( n − 1 ) A_{cancel i, cancel j}in mathbb{R}^{(n-1) imes (n-1)} A i , j ∈ R ( n − 1 ) × ( n − 1 ) i i i j j j
∣ A ∣ = ∑ i = 1 n ( − 1 ) i + j a i j ∣ A i , j ∣ ( for any j ∈ 1 , … , n ) = ∑ j = 1 n ( − 1 ) i + j a i j ∣ A i , j ∣ ( for any i ∈ 1 , … , n ) egin{aligned}|A| &=sum_{i=1}^{n}(-1)^{i+j} a_{i j}left|A_{ackslash i, ackslash j}
ight| quad( ext { for any } j in 1, ldots, n) \ &=sum_{j=1}^{n}(-1)^{i+j} a_{i j}left|A_{ackslash i, ackslash j}
ight| quad( ext { for any } i in 1, ldots, n) end{aligned} ∣ A ∣ = i = 1 ∑ n ( − 1 ) i + j a ij ∣ ∣ A i , j ∣ ∣ ( for any j ∈ 1 , … , n ) = j = 1 ∑ n ( − 1 ) i + j a ij ∣ ∣ A i , j ∣ ∣ ( for any i ∈ 1 , … , n )
对于 A ∈ R 1 × 1 A in mathbb{R}^{1 imes 1} A ∈ R 1 × 1 ∣ A ∣ = a 11 left| A
ight|= a_{11} ∣ A ∣ = a 11 A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n n ! n! n ! n n n 3 × 3 3×3 3 × 3 3 × 3 3×3 3 × 3
∣ [ a 11 ] ∣ = a 11 ∣ [ a 11 a 12 a 21 a 22 ] ∣ = a 11 a 22 − a 12 a 21 ∣ [ a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 ] ∣ = a 11 a 22 a 33 + a 12 a 23 a 31 + a 13 a 21 a 32 − a 11 a 23 a 32 − a 12 a 21 a 33 − a 13 a 22 a 31 egin{aligned}
left|left[a_{11}
ight]
ight| &=a_{11}
\
left|left[egin{array}{ll}{a_{11}} & {a_{12}} \ {a_{21}} & {a_{22}}end{array}
ight]
ight|&=a_{11} a_{22}-a_{12} a_{21}
\
left|left[egin{array}{l}{a_{11}} & {a_{12}} & {a_{13}} \ {a_{21}} & {a_{22}} & {a_{23}} \ {a_{31}} & {a_{32}} & {a_{33}}end{array}
ight]
ight| &=
egin{array}{c}{a_{11} a_{22} a_{33}+a_{12} a_{23} a_{31}+a_{13} a_{21} a_{32}} \ {-a_{11} a_{23} a_{32}-a_{12} a_{21} a_{33}-a_{13} a_{22} a_{31}} end{array}
end{aligned} ∣ [ a 11 ] ∣ ∣ ∣ [ a 11 a 21 a 12 a 22 ] ∣ ∣ ∣ ∣ ⎣ ⎡ a 11 a 21 a 31 a 12 a 22 a 32 a 13 a 23 a 33 ⎦ ⎤ ∣ ∣ = a 11 = a 11 a 22 − a 12 a 21 = a 11 a 22 a 33 + a 12 a 23 a 31 + a 13 a 21 a 32 − a 11 a 23 a 32 − a 12 a 21 a 33 − a 13 a 22 a 31
矩阵A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n 经典伴随矩阵 伴随矩阵 ）表示为adj ( A ) operatorname{adj}(A) adj ( A )
adj ( A ) ∈ R n × n , ( adj ( A ) ) i j = ( − 1 ) i + j ∣ A j , i ∣ operatorname{adj}(A) in mathbb{R}^{n imes n}, quad(operatorname{adj}(A))_{i j}=(-1)^{i+j}left|A_{ackslash j, ackslash i}
ight| adj ( A ) ∈ R n × n , ( adj ( A ) ) ij = ( − 1 ) i + j ∣ ∣ A j , i ∣ ∣
（注意索引A j , i A_{ackslash j, ackslash i} A j , i A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n
A − 1 = 1 ∣ A ∣ adj ( A ) A^{-1}=frac{1}{|A|} operatorname{adj}(A) A − 1 = ∣ A ∣ 1 adj ( A )
虽然这是一个很好的“显式”的逆矩阵公式，但我们应该注意，从数字上讲，有很多更有效的方法来计算逆矩阵。
给定方矩阵A ∈ R n × n A in mathbb{R}^{n imes n} A ∈ R n × n x ∈ R n x in mathbb{R}^{n} x ∈ R n x T A x x^T Ax x T A x 二次型
x T A x = ∑ i = 1 n x i ( A x ) i = ∑ i = 1 n x i ( ∑ j = 1 n A i j x j ) = ∑ i = 1 n ∑ j = 1 n A i j x i x j x^{T} A x=sum_{i=1}^{n} x_{i}(A x)_{i}=sum_{i=1}^{n} x_{i}left(sum_{j=1}^{n} A_{i j} x_{j}
ight)=sum_{i=1}^{n} sum_{j=1}^{n} A_{i j} x_{i} x_{j} x T A x = i = 1 ∑ n x i ( A x ) i = i = 1 ∑ n x i ( j = 1 ∑ n A ij x j ) = i = 1 ∑ n j = 1 ∑ n A ij x i x j
注意：
x T A x = ( x T A x ) T = x T A T x = x T ( 1 2 A + 1 2 A T ) x x^{T} A x=left(x^{T} A x
ight)^{T}=x^{T} A^{T} x=x^{T}left(frac{1}{2} A+frac{1}{2} A^{T}
ight) x x T A x = ( x T A x ) T = x T A T x = x T ( 2 1 A + 2 1 A T ) x
小技巧：
A = A + A T 2 + A − A T 2 , x T A x = x T A + A T 2 x + x T A − A T 2 x = x T A + A T 2 x + 0 A = frac{A+A^T}{2} + frac{A-A^T}{2}\,
\
x^{T} A x = x^{T}frac{A+A^T}{2}x + x^{T}frac{A-A^T}{2}x = x^{T}frac{A+A^T}{2}x +0 A = 2 A + A T + 2 A − A T , x T A x = x T 2 A + A T x + x T 2 A − A T x = x T 2 A + A T x + 0
第一个等号的是因为是标量的转置与自身相等，而第二个等号是因为是我们平均两个本身相等的量。 由此，我们可以得出结论，只有A A A A + A T A+A^T A + A T 对称的 )。 出于这个原因，我们经常隐含地假设以二次型出现的矩阵是对称阵。
对于所有非零向量x ∈ R n x in mathbb{R}^n x ∈ R n x T A x > 0 x^TAx>0 x T A x > 0 A ∈ S n A in mathbb{S}^n A ∈ S n 正定(PD) A ≻ 0 Asucc0 A ≻ 0 A > 0 A>0 A > 0 S + + n mathbb{S}_{++}^n S ++ n
对于所有向量x T A x ≥ 0 x^TAxgeq 0 x T A x ≥ 0 A ∈ S n A in mathbb{S}^n A ∈ S n 半正定(PSD) A ⪰ 0 A succeq 0 A ⪰ 0 A ≥ 0 A≥0 A ≥ 0 S + n mathbb{S}_+^n S + n
同样，对称矩阵A ∈ S n A in mathbb{S}^n A ∈ S n 负定(ND) x ∈ R n x in mathbb{R}^n x ∈ R n x T A x < 0 x^TAx <0 x T A x < 0 A ≺ 0 Aprec0 A ≺ 0 A < 0 A <0 A < 0
类似地，对称矩阵A ∈ S n A in mathbb{S}^n A ∈ S n 半负定(NSD) x ∈ R n x in mathbb{R}^n x ∈ R n x T A x ≤ 0 x^TAx leq 0 x T A x ≤ 0 A ⪯ 0 Apreceq 0 A ⪯ 0 A ≤ 0 A≤0 A ≤ 0
最后，对称矩阵A ∈ S n A in mathbb{S}^n A ∈ S n 不定 x 1 , x 2 ∈ R n x_1,x_2 in mathbb{R}^n x 1 , x 2 ∈ R n x 1 T A x 1 > 0 x_1^TAx_1>0 x 1 T A x 1 > 0 x 2 T A x 2 < 0 x_2^TAx_2<0 x 2 T A x 2 < 0
很明显，如果A A A − A −A − A A A A − A −A − A A A A − A −A − A
正定矩阵和负定矩阵的一个重要性质是它们总是满秩，因此是可逆的。为了了解这是为什么，假设某个矩阵A ∈ S n A in mathbb{S}^n A ∈ S n A A A j j j n − 1 n-1 n − 1
a j = ∑ i ≠ j x i a i a_{j}=sum_{i
eq j} x_{i} a_{i} a j = i � = j ∑ x i a i
对于某些x 1 , ⋯ x j − 1 , x j + 1 , ⋯ , x n ∈ R x_1,cdots x_{j-1},x_{j + 1} ,cdots ,x_nin mathbb{R} x 1 , ⋯ x j − 1 , x j + 1 , ⋯ , x n ∈ R x j = − 1 x_j = -1 x j = − 1
A x = ∑ i = 1 n x i a i = 0 Ax=sum_{i =1}^n x_{i} a_{i}=0 A x = i = 1 ∑ n x i a i = 0
但这意味着对于某些非零向量x x x x T A x = 0 x^T Ax = 0 x T A x = 0 A A A A A A A ∈ R m × n A in mathbb{R}^{m imes n} A ∈ R m × n G = A T A ∈ R n × n G = A^T A in Bbb{R}^{n imes n} G = A T A ∈ R n × n Gram 矩阵 m ≥ n mgeq n m ≥ n A A A rank ( A ) = n operatorname{rank}(A) = n rank ( A ) = n G = A T A G = A^T A G = A T A
A A T AA^T A A T
给定一个方阵A ∈ R n × n A inmathbb{R}^{n imes n} A ∈ R n × n λ ∈ C lambda inmathbb{C} λ ∈ C A A A 特征值 x ∈ C n xinmathbb{C}^n x ∈ C n 特征向量
A x = λ x , x ≠ 0 Ax=lambda x,x
e 0 A x = λ x , x � = 0
直观地说，这个定义意味着将A A A x x x x x x λ lambda λ x ∈ C n xinmathbb{C}^n x ∈ C n c ∈ C cinmathbb{C} c ∈ C A ( c x ) = c A x = c λ x = λ ( c x ) A(cx)=cAx=clambda x=lambda(cx) A ( c x ) = c A x = c λ x = λ ( c x ) c x cx c x λ lambda λ x x x − x −x − x
我们可以重写上面的等式来说明( λ , x ) (lambda,x) ( λ , x ) A A A
( λ I − A ) x = 0 , x ≠ 0 (lambda I-A)x=0,x
e 0 ( λ I − A ) x = 0 , x � = 0
但是( λ I − A ) x = 0 (lambda I-A)x=0 ( λ I − A ) x = 0 ( λ I − A ) (lambda I-A) ( λ I − A ) ( λ I − A ) (lambda I-A) ( λ I − A ) x x x
∣ ( λ I − A ) ∣ = 0 |(lambda I-A)|=0 ∣ ( λ I − A ) ∣ = 0
我们现在可以使用之前的行列式定义来展开这个表达式式∣ ( λ I − A ) ∣ |(lambda I-A)| ∣ ( λ I − A ) ∣ λ lambda λ λ lambda λ n n n A A A
然后我们找到这个特征多项式的n n n λ 1 , ⋯ , λ n lambda_1,cdots,lambda_n λ 1 , ⋯ , λ n A A A λ i lambda_i λ i ( λ I − A ) x = 0 (lambda I-A)x=0 ( λ I − A ) x = 0 ( λ I − A ) (lambda I-A) ( λ I − A )
应该注意的是，这不是实际用于数值计算特征值和特征向量的方法（记住行列式的完全展开式有n ! n! n !
以下是特征值和特征向量的性质（所有假设在A ∈ R n × n A inmathbb{R}^{n imes n} A ∈ R n × n λ 1 , ⋯ , λ n lambda_1,cdots,lambda_n λ 1 , ⋯ , λ n
A A A
tr A = ∑ i = 1 n λ i = ∑ i = 1 n A i i operatorname{tr} A=sum_{i=1}^{n} lambda_{i} =sum_{i=1}^{n} A_{ii} tr A = i = 1 ∑ n λ i = i = 1 ∑ n A ii
A A A
∣ A ∣ = ∏ i = 1 n λ i |A|=prod_{i=1}^{n} lambda_{i} ∣ A ∣ = i = 1 ∏ n λ i
A A A A A A
假设A A A λ lambda λ x x x 1 / λ 1/lambda 1/ λ x x x A − 1 A^{-1} A − 1 A − 1 x = ( 1 / λ ) x A^{-1}x=(1/lambda)x A − 1 x = ( 1/ λ ) x A x = λ x Ax=lambda x A x = λ x A − 1 A^{-1} A − 1
对角阵的特征值D = diag ( d 1 , ⋯ d n ) D=operatorname{diag}(d_1,cdots d_n) D = diag ( d 1 , ⋯ d n ) d 1 , ⋯ d n d_1,cdots d_n d 1 , ⋯ d n
一般而言，一般方阵的特征值和特征向量的结构很难表征。 幸运的是，在机器学习的大多数情况下，处理对称实矩阵就足够了，其特征值和特征向量具有显着的性质。
在本节中，我们假设A A A
A A A λ 1 , ⋯ , λ n lambda_1,cdots,lambda_n λ 1 , ⋯ , λ n
存在一组特征向量u 1 , ⋯ , u n u_1,cdots,u_n u 1 , ⋯ , u n i i i u i u_i u i λ i lambda_{i} λ i u 1 , ⋯ , u n u_1,cdots,u_n u 1 , ⋯ , u n
设U U U u i u_i u i
U = [ ∣ ∣ ∣ u 1 u 2 ⋯ u n ∣ ∣ ∣ ] (5) U=left[egin{array}{cccc}{ |} & { |} & {} & { |} \ {u_{1}} & {u_{2}} & {cdots} & {u_{n}} \ { |} & { |} & {} & { |}end{array}
ight]
label{5} ag{5} U = ⎣ ⎡ ∣ u 1 ∣ ∣ u 2 ∣ ⋯ ∣ u n ∣ ⎦ ⎤ ( 5 )
设Λ = diag ( λ 1 , ⋯ , λ n ) Lambda= operatorname{diag}(lambda_1,cdots,lambda_n) Λ = diag ( λ 1 , ⋯ , λ n ) λ 1 , ⋯ , λ n lambda_1,cdots,lambda_n λ 1 , ⋯ , λ n ( 2 ) eqref{2} ( 2 )
A U = [ ∣ ∣ ∣ A u 1 A u 2 ⋯ A u n ∣ ∣ ∣ ] = [ ∣ ∣ ∣ λ 1 u 1 λ 2 u 2 ⋯ λ n u n ∣ ∣ ∣ ] = U diag ( λ 1 , … , λ n ) = U Λ A U=left[egin{array}{cccc}
{ |} & { |} & {} & { |} \
{A u_{1}} & {A u_{2}} & {cdots} & {A u_{n}} \
{ |} & { |} & {} & { |}end{array}
ight]=
left[egin{array}{ccc}
{ |} & { |} & { } & { |}\
{lambda_{1} u_{1}} & {lambda_{2} u_{2}} & {cdots} & {lambda_{n} u_{n}} \
{ |} & { |} & {} & { |}
end{array}
ight]=
U operatorname{diag}left(lambda_{1}, ldots, lambda_{n}
ight)=U Lambda A U = ⎣ ⎡ ∣ A u 1 ∣ ∣ A u 2 ∣ ⋯ ∣ A u n ∣ ⎦ ⎤ = ⎣ ⎡ ∣ λ 1 u 1 ∣ ∣ λ 2 u 2 ∣ ⋯ ∣ λ n u n ∣ ⎦ ⎤ = U diag ( λ 1 , … , λ n ) = U Λ
考虑到正交矩阵U U U U U T = I UU^T=I U U T = I
A = A U U T = U Λ U T (6) A=AUU^T=ULambda U^T
label{6} ag{6} A = A U U T = U Λ U T ( 6 )
这种A A A U Λ U T ULambda U^T U Λ U T A A A 对角化 A A A U U U
背景知识：关于另一个基的向量 。
任何正交矩阵U = [ ∣ ∣ ∣ u 1 u 2 ⋯ u n ∣ ∣ ∣ ] U=left[egin{array}{cccc}{ |} & { |} & {} & { |} \ {u_{1}} & {u_{2}} & {cdots} & {u_{n}} \ { |} & { |} & {} & { |}end{array}
ight] U = ⎣ ⎡ ∣ u 1 ∣ ∣ u 2 ∣ ⋯ ∣ u n ∣ ⎦ ⎤ R n mathbb {R}^{n} R n x ∈ R n x inmathbb{R}^{n} x ∈ R n u 1 , ⋯ , u n u_1,cdots,u_n u 1 , ⋯ , u n x ^ 1 , ⋯ , x ^ n hat x_1,cdots,hat x_n x ^ 1 , ⋯ , x ^ n
x = x ^ 1 u 1 + ⋯ + x ^ n u n = U x ^ x=hat x_1u_1+cdots + hat x_nu_n=Uhat x x = x ^ 1 u 1 + ⋯ + x ^ n u n = U x ^
在第二个等式中，我们使用矩阵和向量相乘的方法,查看式( 1 ) eqref{1} ( 1 ) x ^ hat x x ^
x = U x ^ ⇔ U T x = x ^ x=U hat{x} Leftrightarrow U^{T} x=hat{x} x = U x ^ ⇔ U T x = x ^
换句话说，向量x ^ = U T x hat x=U^Tx x ^ = U T x x x x U U U
“对角化”矩阵向量乘法 。 通过上面的设置，我们将看到左乘矩阵A A A x x x x ^ hat x x ^ U U U x x x z = A x z=Ax z = A x U U U z z z U U T = U T U = I UU^T=U^TU=I U U T = U T U = I A = A U U T = U Λ U T A=AUU^T=ULambda U^T A = A U U T = U Λ U T ( 6 ) eqref{6} ( 6 )
z ^ = U T z = U T A x = U T U Λ U T x = Λ x ^ = [ λ 1 x ^ 1 λ 2 x ^ 2 ⋮ λ n x ^ n ] hat{z}=U^{T} z=U^{T} A x=U^{T} U Lambda U^{T} x=Lambda hat{x}=left[egin{array}{c}{lambda_{1} hat{x}_{1}} \ {lambda_{2} hat{x}_{2}} \ {vdots} \ {lambda_{n} hat{x}_{n}}end{array}
ight] z ^ = U T z = U T A x = U T U Λ U T x = Λ x ^ = ⎣ ⎡ λ 1 x ^ 1 λ 2 x ^ 2 ⋮ λ n x ^ n ⎦ ⎤
我们可以看到，原始空间中的左乘矩阵A A A Λ Lambda Λ q = A A A x q=AAAx q = AAA x A A A q q q
q ^ = U T q = U T A A A x = U T U Λ U T U Λ U T U Λ U T x = Λ 3 x ^ = [ λ 1 3 x ^ 1 λ 2 3 x ^ 2 ⋮ λ n 3 x ^ n ] (7) hat{q}=
U^{T} q=
U^{T} AAA x=
U^{T} U Lambda U^{T} U Lambda U^{T} U Lambda U^{T} x=
Lambda^{3} hat{x}=
left[egin{array}{c}
{lambda_{1}^{3} hat{x}_{1}} \ {lambda_{2}^{3} hat{x}_{2}} \ {vdots} \ {lambda_{n}^{3} hat{x}_{n}}
end{array}
ight]
label{7} ag{7} q ^ = U T q = U T AAA x = U T U Λ U T U Λ U T U Λ U T x = Λ 3 x ^ = ⎣ ⎡ λ 1 3 x ^ 1 λ 2 3 x ^ 2 ⋮ λ n 3 x ^ n ⎦ ⎤ ( 7 )
“对角化”二次型 。作为直接推论，二次型x T A x x^TAx x T A x
x T A x = x T U Λ U T x = x ^ T Λ x ^ = ∑ i = 1 n λ i x ^ i 2 (8) x^{T} A x=x^{T} U Lambda U^{T} x=hat{x}^T Lambda hat{x}=sum_{i=1}^{n} lambda_{i} hat{x}_{i}^{2}
label{8} ag{8} x T A x = x T U Λ U T x = x ^ T Λ x ^ = i = 1 ∑ n λ i x ^ i 2 ( 8 )
(回想一下，在旧的表示法中，x T A x = ∑ i = 1 , j = 1 n x i x j A i j x^{T} A x=sum_{i=1, j=1}^{n} x_{i} x_{j} A_{i j} x T A x = ∑ i = 1 , j = 1 n x i x j A ij n 2 n^2 n 2 n n n A A A
如果所有的λ i > 0 lambda_i>0 λ i > 0 A A A x ^ ≠ 0 hat x
e 0 x ^ � = 0 x T A x = ∑ i = 1 n λ i x ^ i 2 > 0 x^{T} A x=sum_{i=1}^{n} lambda_{i} hat{x}_{i}^{2}>0 x T A x = ∑ i = 1 n λ i x ^ i 2 > 0
如果所有的λ i ≥ 0 lambda_igeq 0 λ i ≥ 0 A A A x ^ hat x x ^ x T A x = ∑ ∗ i = 1 n λ ∗ i x ^ _ i 2 ≥ 0 x^{T} A x=sum*{i=1}^{n} lambda*{i} hat{x}\_{i}^{2} geq 0 x T A x = ∑ ∗ i = 1 n λ ∗ i x ^ _ i 2 ≥ 0
同样，如果所有λ i < 0 lambda_i<0 λ i < 0 λ i ≤ 0 lambda_ileq 0 λ i ≤ 0 A A A
最后，如果A A A λ i > 0 lambda_i>0 λ i > 0 λ j < 0 lambda_j<0 λ j < 0 x ^ hat x x ^ x ^ i = 1 and x ^ k = 0 , ∀ k ≠ i hat x_i=1 ext{ and } hat x_k=0, forall k
e i x ^ i = 1 and x ^ k = 0 , ∀ k � = i x T A x = ∑ i = 1 n λ i x ^ i 2 > 0 x^{T} A x=sum_{i=1}^{n} lambda_{i} hat{x}_{i}^{2}>0 x T A x = ∑ i = 1 n λ i x ^ i 2 > 0 x ^ hat x x ^ x ^ j = 1 and x ^ k = 0 , ∀ k ≠ j hat x_j=1 ext{ and } hat x_k=0,forall k
e j x ^ j = 1 and x ^ k = 0 , ∀ k � = j x T A x = ∑ i = 1 n λ i x ^ i 2 < 0 x^{T} A x=sum_{i=1}^{n} lambda_{i} hat{x}_{i}^{2}<0 x T A x = ∑ i = 1 n λ i x ^ i 2 < 0
特征值和特征向量经常出现的应用是最大化矩阵的某些函数。特别是对于矩阵A ∈ S n A in mathbb{S}^{n} A ∈ S n
max x ∈ R n x T A x = ∑ i = 1 n λ i x ^ i 2 subject to ∥ x ∥ 2 2 = 1 (9) max _{x in mathbb{R}^{n}} x^{T} A x=sum_{i=1}^{n} lambda_{i} hat{x}_{i}^{2} quad ext { subject to }|x|_{2}^{2}=1
label{9} ag{9} x ∈ R n max x T A x = i = 1 ∑ n λ i x ^ i 2 subject to ∥ x ∥ 2 2 = 1 ( 9 )
也就是说，我们要找到（范数 1）的向量，它使二次型最大化。假设特征值的阶数为λ 1 ≥ λ 2 ≥ ⋯ λ n lambda_1 geq lambda _2 geq cdots lambda_n λ 1 ≥ λ 2 ≥ ⋯ λ n λ 1 lambda_1 λ 1 λ 1 lambda_1 λ 1 u 1 u_1 u 1 λ 1 > λ 2 lambda_1 > lambda_2 λ 1 > λ 2 λ 1 lambda_1 λ 1 ( 9 ) eqref{9} ( 9 )
我们可以通过使用对角化技术来证明这一点：注意，通过公式∥ U x ∥ 2 = ( 3 ) ∥ x ∥ 2 |U x|_{2}overset{eqref{3}}{=}|x|_{2} ∥ Ux ∥ 2 = ( 3 ) ∥ x ∥ 2 ∥ x ∥ 2 = ∥ x ^ ∥ 2 |x|_{2}=|hat{x}|_{2} ∥ x ∥ 2 = ∥ x ^ ∥ 2 x T A x = x T U Λ U T x = x ^ T Λ x ^ = ∑ ∗ i = 1 n λ ∗ i x ^ _ i 2 , 也就是式 ( 8 ) x^{T} A x=x^{T} U Lambda U^{T} x=hat{x}^T Lambda hat{x}=sum*{i=1}^{n} lambda*{i} hat{x}\_{i}^{2} , ext{也就是式}eqref{8} x T A x = x T U Λ U T x = x ^ T Λ x ^ = ∑ ∗ i = 1 n λ ∗ i x ^ _ i 2 , 也就是式 ( 8 )
max x ^ ∈ R n x ^ T Λ x ^ = ∑ i = 1 n λ i x ^ i 2 subject to ∥ x ^ ∥ 2 2 = 1 max _{hat{x} in mathbb{R}^{n}} hat{x}^{T} Lambda hat{x}=sum_{i=1}^{n} lambda_{i} hat{x}_{i}^{2} quad ext { subject to }|hat{x}|_{2}^{2}=1 x ^ ∈ R n max x ^ T Λ x ^ = i = 1 ∑ n λ i x ^ i 2 subject to ∥ x ^ ∥ 2 2 = 1
然后，我们得到目标的上界为λ 1 lambda_1 λ 1
x ^ T Λ x ^ = ∑ i = 1 n λ i x ^ i 2 ≤ ∑ i = 1 n λ 1 x ^ i 2 = λ 1 hat{x}^{T} Lambda hat{x}=sum_{i=1}^{n} lambda_{i} hat{x}_{i}^{2} leq sum_{i=1}^{n} lambda_{1} hat{x}_{i}^{2}=lambda_{1} x ^ T Λ x ^ = i = 1 ∑ n λ i x ^ i 2 ≤ i = 1 ∑ n λ 1 x ^ i 2 = λ 1
此外，设置x ^ = [ 1 0 ⋮ 0 ] hat{x}=left[egin{array}{c}{1} \ {0} \ {vdots} \ {0}end{array}
ight] x ^ = ⎣ ⎡ 1 0 ⋮ 0 ⎦ ⎤ x = u 1 x=u_1 x = u 1
虽然前几节中的主题通常在线性代数的标准课程中涵盖，但一个似乎不经常涉及（我们将广泛使用）的主题是微积分对向量设置的扩展。 尽管我们使用的所有实际微积分都相对微不足道，但符号通常会使事情看起来比实际困难得多。 在本节中，我们将介绍矩阵微积分的一些基本定义并提供一些示例。
假设f : R m × n → R f: mathbb{R}^{m imes n}
ightarrow mathbb{R} f : R m × n → R m × n m imes n m × n A ∈ R m × n Ain mathbb{R}^{m imes n} A ∈ R m × n f f f 梯度 A ∈ R m × n Ain mathbb{R}^{m imes n} A ∈ R m × n
∇ A f ( A ) ∈ R m × n = [ ∂ f ( A ) ∂ A 11 ∂ f ( A ) ∂ A 12 ⋯ ∂ f ( A ) ∂ A 1 n ∂ f ( A ) ∂ A 21 ∂ f ( A ) ∂ A 22 ⋯ ∂ f ( A ) ∂ A 2 n ⋮ ⋮ ⋱ ⋮ ∂ f ( A ) ∂ A m 1 ∂ f ( A ) ∂ A m 2 ⋯ ∂ f ( A ) ∂ A m n ]
abla_{A} f(A) in mathbb{R}^{m imes n}=left[egin{array}{cccc}{frac{partial f(A)}{partial A_{11}}} & {frac{partial f(A)}{partial A_{12}}} & {cdots} & {frac{partial f(A)}{partial A_{1n}}} \ {frac{partial f(A)}{partial A_{21}}} & {frac{partial f(A)}{partial A_{22}}} & {cdots} & {frac{partial f(A)}{partial A_{2 n}}} \ {vdots} & {vdots} & {ddots} & {vdots} \ {frac{partial f(A)}{partial A_{m 1}}} & {frac{partial f(A)}{partial A_{m 2}}} & {cdots} & {frac{partial f(A)}{partial A_{m n}}}end{array}
ight] ∇ A f ( A ) ∈ R m × n = ⎣ ⎡ ∂ A 11 ∂ f ( A ) ∂ A 21 ∂ f ( A ) ⋮ ∂ A m 1 ∂ f ( A ) ∂ A 12 ∂ f ( A ) ∂ A 22 ∂ f ( A ) ⋮ ∂ A m 2 ∂ f ( A ) ⋯ ⋯ ⋱ ⋯ ∂ A 1 n ∂ f ( A ) ∂ A 2 n ∂ f ( A ) ⋮ ∂ A mn ∂ f ( A ) ⎦ ⎤
即，m × n m imes n m × n
( ∇ A f ( A ) ) i j = ∂ f ( A ) ∂ A i j left(
abla_{A} f(A)
ight)_{i j}=frac{partial f(A)}{partial A_{i j}} ( ∇ A f ( A ) ) ij = ∂ A ij ∂ f ( A )
请注意 ，∇ A f ( A )
abla_{A} f(A) ∇ A f ( A ) A A A A A A A ∈ R n Ain mathbb{R}^{n} A ∈ R n
∇ x f ( x ) = [ ∂ f ( x ) ∂ x 1 ∂ f ( x ) ∂ x 2 ⋮ ∂ f ( x ) ∂ x n ]
abla_{x} f(x)=left[egin{array}{c}{frac{partial f(x)}{partial x_{1}}} \ {frac{partial f(x)}{partial x_{2}}} \ {vdots} \ {frac{partial f(x)}{partial x_{n}}}end{array}
ight] ∇ x f ( x ) = ⎣ ⎡ ∂ x 1 ∂ f ( x ) ∂ x 2 ∂ f ( x ) ⋮ ∂ x n ∂ f ( x ) ⎦ ⎤
重要的是要记住，只有当函数是实值时，即如果函数返回标量值，才定义函数的梯度。例如，A ∈ R m × n Ain mathbb{R}^{m imes n} A ∈ R m × n x x x A x Ax A x
直接从偏导数的等价性质得出：
∇ x ( f ( x ) + g ( x ) ) = ∇ x f ( x ) + ∇ x g ( x )
abla_{x}(f(x)+g(x))=
abla_{x} f(x)+
abla_{x} g(x) ∇ x ( f ( x ) + g ( x )) = ∇ x f ( x ) + ∇ x g ( x )
For t ∈ R , ∇ x ( t f ( x ) ) = t ∇ x f ( x ) ext{For }t in mathbb{R},
abla_{x}(t f(x))=t
abla_{x} f(x) For t ∈ R , ∇ x ( t f ( x )) = t ∇ x f ( x )
原则上，梯度是偏导数对多维变量函数的自然延伸。然而，在实践中，由于符号的原因，使用梯度有时是很棘手的。例如，假设A ∈ R m × n Ain mathbb{R}^{m imes n} A ∈ R m × n b ∈ R m bin mathbb{R}^{m} b ∈ R m f : R m → R f: mathbb{R}^{m}
ightarrow mathbb{R} f : R m → R f ( z ) = z T z f(z)=z^Tz f ( z ) = z T z ∇ z f ( z ) = 2 z
abla_{z}f(z)=2z ∇ z f ( z ) = 2 z
∇ f ( A x )
abla f(Ax) ∇ f ( A x )
该表达式应该如何解释？ 至少有两种可能性：
在第一个解释中，回想起∇ z f ( z ) = 2 z
abla_{z}f(z)=2z ∇ z f ( z ) = 2 z ∇ f ( A x )
abla f(Ax) ∇ f ( A x ) A x Ax A x
∇ f ( A x ) = 2 ( A x ) = 2 A x ∈ R m
abla f(A x)=2(A x)=2 A x in mathbb{R}^{m} ∇ f ( A x ) = 2 ( A x ) = 2 A x ∈ R m
在第二种解释中，我们将数量f ( A x ) f(Ax) f ( A x ) x x x g ( x ) = f ( A x ) g(x) =f(Ax) g ( x ) = f ( A x )
∇ f ( A x ) = ∇ x g ( x ) ∈ R n
abla f(A x)=
abla_{x} g(x) in mathbb{R}^{n} ∇ f ( A x ) = ∇ x g ( x ) ∈ R n
在这里，我们可以看到这两种解释确实不同。 一种解释产生m m m n n n x x x n n n ∇ x g ( x )
abla_{x} g(x) ∇ x g ( x ) n n n
这里，关键是要明确我们要区分的变量。f f f z z z A x Ax A x g ( x ) = f ( A x ) g(x)=f(Ax) g ( x ) = f ( A x ) x x x
我们将第一种情况表示为∇ z f ( A x )
abla zf(Ax) ∇ z f ( A x ) ∇ x f ( A x )
abla xf(Ax) ∇ x f ( A x )
保持符号清晰是非常重要的，以后完成课程作业时候你就会发现。
假设f : R n → R f: mathbb{R}^{n}
ightarrow mathbb{R} f : R n → R R n mathbb{R}^{n} R n x x x 黑塞矩阵 ∇ x 2 f ( A x )
abla_x ^2 f(A x) ∇ x 2 f ( A x ) H H H n × n n imes n n × n
∇ x 2 f ( x ) ∈ R n × n = [ ∂ 2 f ( x ) ∂ x 1 2 ∂ 2 f ( x ) ∂ x 1 ∂ x 2 ⋯ ∂ 2 f ( x ) ∂ x 1 ∂ x n ∂ 2 f ( x ) ∂ x 2 ∂ x 1 ∂ 2 f ( x ) ∂ x 2 2 ⋯ ∂ 2 f ( x ) ∂ x 2 ∂ x n ⋮ ⋮ ⋱ ⋮ ∂ 2 f ( x ) ∂ x n ∂ x 1 ∂ 2 f ( x ) ∂ x n ∂ x 2 ⋯ ∂ 2 f ( x ) ∂ x n 2 ]
abla_{x}^{2} f(x) in mathbb{R}^{n imes n}=left[egin{array}{cccc}{frac{partial^{2} f(x)}{partial x_{1}^{2}}} & {frac{partial^{2} f(x)}{partial x_{1} partial x_{2}}} & {cdots} & {frac{partial^{2} f(x)}{partial x_{1} partial x_{n}}} \ {frac{partial^{2} f(x)}{partial x_{2} partial x_{1}}} & {frac{partial^{2} f(x)}{partial x_{2}^{2}}} & {cdots} & {frac{partial^{2} f(x)}{partial x_{2} partial x_{n}}} \ {vdots} & {vdots} & {ddots} & {vdots} \ {frac{partial^{2} f(x)}{partial x_{n} partial x_{1}}} & {frac{partial^{2} f(x)}{partial x_{n} partial x_{2}}} & {cdots} & {frac{partial^{2} f(x)}{partial x_{n}^{2}}}end{array}
ight] ∇ x 2 f ( x ) ∈ R n × n = ⎣ ⎡ ∂ x 1 2 ∂ 2 f ( x ) ∂ x 2 ∂ x 1 ∂ 2 f ( x ) ⋮ ∂ x n ∂ x 1 ∂ 2 f ( x ) ∂ x 1 ∂ x 2 ∂ 2 f ( x ) ∂ x 2 2 ∂ 2 f ( x ) ⋮ ∂ x n ∂ x 2 ∂ 2 f ( x ) ⋯ ⋯ ⋱ ⋯ ∂ x 1 ∂ x n ∂ 2 f ( x ) ∂ x 2 ∂ x n ∂ 2 f ( x ) ⋮ ∂ x n 2 ∂ 2 f ( x ) ⎦ ⎤
换句话说，∇ x 2 f ( x ) ∈ R n × n
abla_{x}^{2} f(x) in mathbb{R}^{n imes n} ∇ x 2 f ( x ) ∈ R n × n
( ∇ x 2 f ( x ) ) i j = ∂ 2 f ( x ) ∂ x i ∂ x j left(
abla_{x}^{2} f(x)
ight)_{i j}=frac{partial^{2} f(x)}{partial x_{i} partial x_{j}} ( ∇ x 2 f ( x ) ) ij = ∂ x i ∂ x j ∂ 2 f ( x )
注意：黑塞矩阵通常是对称阵：
∂ 2 f ( x ) ∂ x i ∂ x j = ∂ 2 f ( x ) ∂ x j ∂ x i frac{partial^{2} f(x)}{partial x_{i} partial x_{j}}=frac{partial^{2} f(x)}{partial x_{j} partial x_{i}} ∂ x i ∂ x j ∂ 2 f ( x ) = ∂ x j ∂ x i ∂ 2 f ( x )
与梯度相似，只有当f ( x ) f(x) f ( x )
很自然地认为梯度与向量函数的一阶导数的相似，而黑塞矩阵与二阶导数的相似（我们使用的符号也暗示了这种关系）。 这种直觉通常是正确的，但需要记住以下几个注意事项。f : R → R f: mathbb{R}
ightarrow mathbb{R} f : R → R
∂ 2 f ( x ) ∂ x 2 = ∂ ∂ x ∂ ∂ x f ( x ) frac{partial^{2} f(x)}{partial x^{2}}=frac{partial}{partial x} frac{partial}{partial x} f(x) ∂ x 2 ∂ 2 f ( x ) = ∂ x ∂ ∂ x ∂ f ( x )
然而，对于向量的函数，函数的梯度是一个向量，我们不能取向量的梯度，即:
∇ x ∇ x f ( x ) = ∇ x [ ∂ f ( x ) ∂ x 1 ∂ f ( x ) ∂ x 2 ⋮ ∂ f ( x ) ∂ x n ]
abla_{x}
abla_{x} f(x)=
abla_{x}left[egin{array}{c}{frac{partial f(x)}{partial x_{1}}} \ {frac{partial f(x)}{partial x_{2}}} \ {vdots} \ {frac{partial f(x)}{partial x_{n}}}end{array}
ight] ∇ x ∇ x f ( x ) = ∇ x ⎣ ⎡ ∂ x 1 ∂ f ( x ) ∂ x 2 ∂ f ( x ) ⋮ ∂ x n ∂ f ( x ) ⎦ ⎤
上面这个表达式没有意义。 因此，黑塞矩阵不是梯度的梯度。 然而，下面这种情况却这几乎是正确的：如果我们看一下梯度( ∇ x f ( x ) ) i = ∂ f ( x ) / ∂ x i left(
abla_{x} f(x)
ight)_{i}=partial f(x) / partial x_{i} ( ∇ x f ( x ) ) i = ∂ f ( x ) / ∂ x i i i i x x x
∇ x ∂ f ( x ) ∂ x i = [ ∂ 2 f ( x ) ∂ x i ∂ x 1 ∂ 2 f ( x ) ∂ x 2 ∂ x 2 ⋮ ∂ f ( x ) ∂ x i ∂ x n ]
abla_{x} frac{partial f(x)}{partial x_{i}}=left[egin{array}{c}{frac{partial^{2} f(x)}{partial x_{i} partial x_{1}}} \ {frac{partial^{2} f(x)}{partial x_{2} partial x_{2}}} \ {vdots} \ {frac{partial f(x)}{partial x_{i} partial x_{n}}}end{array}
ight] ∇ x ∂ x i ∂ f ( x ) = ⎣ ⎡ ∂ x i ∂ x 1 ∂ 2 f ( x ) ∂ x 2 ∂ x 2 ∂ 2 f ( x ) ⋮ ∂ x i ∂ x n ∂ f ( x ) ⎦ ⎤
这是黑塞矩阵第i i i
∇ x 2 f ( x ) = [ ∇ x ( ∇ x f ( x ) ) 1 ∇ x ( ∇ x f ( x ) ) 2 ⋯ ∇ x ( ∇ x f ( x ) ) n ]
abla_{x}^{2} f(x)=left[
abla_{x}left(
abla_{x} f(x)
ight)_{1} quad
abla_{x}left(
abla_{x} f(x)
ight)_{2} quad cdots quad
abla_{x}left(
abla_{x} f(x)
ight)_{n}
ight] ∇ x 2 f ( x ) = [ ∇ x ( ∇ x f ( x ) ) 1 ∇ x ( ∇ x f ( x ) ) 2 ⋯ ∇ x ( ∇ x f ( x ) ) n ]
简单地说：我们可以说由于：∇ x 2 f ( x ) = ∇ x ( ∇ x f ( x ) ) T
abla_{x}^{2} f(x)=
abla_{x}left(
abla_{x} f(x)
ight)^{T} ∇ x 2 f ( x ) = ∇ x ( ∇ x f ( x ) ) T ∇ x f ( x )
abla_{x} f(x) ∇ x f ( x )
最后，请注意，虽然我们可以对矩阵A ∈ R n Ain mathbb{R}^{n} A ∈ R n x ∈ R n x in mathbb{R}^{n} x ∈ R n ∂ 2 f ( A ) / ( ∂ A i j ∂ A k ℓ ) partial^{2} f(A) /left(partial A_{i j} partial A_{k ell}
ight) ∂ 2 f ( A ) / ( ∂ A ij ∂ A k ℓ )
现在让我们尝试确定几个简单函数的梯度和黑塞矩阵。 应该注意的是，这里给出的所有梯度都是CS229 讲义中给出的梯度的特殊情况。
对于x ∈ R n x in mathbb{R}^{n} x ∈ R n f ( x ) = b T x f(x)=b^Tx f ( x ) = b T x b ∈ R n b in mathbb{R}^{n} b ∈ R n
f ( x ) = ∑ i = 1 n b i x i f(x)=sum_{i=1}^{n} b_{i} x_{i} f ( x ) = i = 1 ∑ n b i x i
所以：
∂ f ( x ) ∂ x k = ∂ ∂ x k ∑ i = 1 n b i x i = b k frac{partial f(x)}{partial x_{k}}=frac{partial}{partial x_{k}} sum_{i=1}^{n} b_{i} x_{i}=b_{k} ∂ x k ∂ f ( x ) = ∂ x k ∂ i = 1 ∑ n b i x i = b k
由此我们可以很容易地看出∇ x b T x = b
abla_{x} b^{T} x=b ∇ x b T x = b ∂ / ( ∂ x ) a x = a partial /(partial x) a x=a ∂ / ( ∂ x ) a x = a A ∈ S n Ain mathbb{S}^{n} A ∈ S n f ( x ) = x T A x f(x)=x^TAx f ( x ) = x T A x
f ( x ) = ∑ i = 1 n ∑ j = 1 n A i j x i x j f(x)=sum_{i=1}^{n} sum_{j=1}^{n} A_{i j} x_{i} x_{j} f ( x ) = i = 1 ∑ n j = 1 ∑ n A ij x i x j
为了取偏导数，我们将分别考虑包括x k x_k x k x 2 k x_2^k x 2 k
∂ f ( x ) ∂ x k = ∂ ∂ x k ∑ i = 1 n ∑ j = 1 n A i j x i x j = ∂ ∂ x k [ ∑ i ≠ k ∑ j ≠ k A i j x i x j + ∑ i ≠ k A i k x i x k + ∑ j ≠ k A k j x k x j + A k k x k 2 ] = ∑ i ≠ k A i k x i + ∑ j ≠ k A k j x j + 2 A k k x k = ∑ i = 1 n A i k x i + ∑ j = 1 n A k j x j = 2 ∑ i = 1 n A k i x i egin{aligned} frac{partial f(x)}{partial x_{k}} &=frac{partial}{partial x_{k}} sum_{i=1}^{n} sum_{j=1}^{n} A_{i j} x_{i} x_{j} \ &=frac{partial}{partial x_{k}}left[sum_{i
eq k} sum_{j
eq k} A_{i j} x_{i} x_{j}+sum_{i
eq k} A_{i k} x_{i} x_{k}+sum_{j
eq k} A_{k j} x_{k} x_{j}+A_{k k} x_{k}^{2}
ight] \ &=sum_{i
eq k} A_{i k} x_{i}+sum_{j
eq k} A_{k j} x_{j}+2 A_{k k} x_{k} \ &=sum_{i=1}^{n} A_{i k} x_{i}+sum_{j=1}^{n} A_{k j} x_{j}=2 sum_{i=1}^{n} A_{k i} x_{i} end{aligned} ∂ x k ∂ f ( x ) = ∂ x k ∂ i = 1 ∑ n j = 1 ∑ n A ij x i x j = ∂ x k ∂ ⎣ ⎡ i � = k ∑ j � = k ∑ A ij x i x j + i � = k ∑ A ik x i x k + j � = k ∑ A kj x k x j + A kk x k 2 ⎦ ⎤ = i � = k ∑ A ik x i + j � = k ∑ A kj x j + 2 A kk x k = i = 1 ∑ n A ik x i + j = 1 ∑ n A kj x j = 2 i = 1 ∑ n A ki x i
最后一个等式，是因为A A A ∇ x f ( x )
abla_{x} f(x) ∇ x f ( x ) k k k A A A x x x k k k ∇ x x T A x = 2 A x
abla_{x} x^{T} A x=2 A x ∇ x x T A x = 2 A x ∂ / ( ∂ x ) a x 2 = 2 a x partial /(partial x) a x^{2}=2 a x ∂ / ( ∂ x ) a x 2 = 2 a x
最后，让我们来看看二次函数f ( x ) = x T A x f(x)=x^TAx f ( x ) = x T A x b T x b^Tx b T x
∂ 2 f ( x ) ∂ x k ∂ x ℓ = ∂ ∂ x k [ ∂ f ( x ) ∂ x ℓ ] = ∂ ∂ x k [ 2 ∑ i = 1 n A ℓ i x i ] = 2 A ℓ k = 2 A k ℓ frac{partial^{2} f(x)}{partial x_{k} partial x_{ell}}=frac{partial}{partial x_{k}}left[frac{partial f(x)}{partial x_{ell}}
ight]=frac{partial}{partial x_{k}}left[2 sum_{i=1}^{n} A_{ell i} x_{i}
ight]=2 A_{ell k}=2 A_{k ell} ∂ x k ∂ x ℓ ∂ 2 f ( x ) = ∂ x k ∂ [ ∂ x ℓ ∂ f ( x ) ] = ∂ x k ∂ [ 2 i = 1 ∑ n A ℓ i x i ] = 2 A ℓ k = 2 A k ℓ
因此，应该很清楚∇ x 2 x T A x = 2 A
abla_{x}^2 x^{T} A x=2 A ∇ x 2 x T A x = 2 A ∂ 2 / ( ∂ x 2 ) a x 2 = 2 a partial^2 /(partial x^2) a x^{2}=2a ∂ 2 / ( ∂ x 2 ) a x 2 = 2 a
简要概括起来：
∇ x b T x = b
abla_{x} b^{T} x=b ∇ x b T x = b
∇ x x T A x = 2 A x
abla_{x} x^{T} A x=2 A x ∇ x x T A x = 2 A x A A A
∇ x 2 x T A x = 2 A
abla_{x}^2 x^{T} A x=2 A ∇ x 2 x T A x = 2 A A A A
让我们应用上一节中得到的方程来推导最小二乘方程。假设我们得到矩阵A ∈ R m × n Ain mathbb{R}^{m imes n} A ∈ R m × n A A A b ∈ R m bin mathbb{R}^{m} b ∈ R m b ∉ R ( A ) b
otin mathcal{R}(A) b ∈ / R ( A ) x ∈ R n xin mathbb{R}^{n} x ∈ R n A x = b Ax = b A x = b x x x A x Ax A x b b b ∥ A x − b ∥ _ 2 2 |A x-b|\_{2}^{2} ∥ A x − b ∥_ 2 2
使用公式∥ x ∥ 2 = x T x |x|^{2}=x^Tx ∥ x ∥ 2 = x T x
∥ A x − b ∥ 2 2 = ( A x − b ) T ( A x − b ) = x T A T A x − 2 b T A x + b T b egin{aligned}|A x-b|_{2}^{2} &=(A x-b)^{T}(A x-b) \ &=x^{T} A^{T} A x-2 b^{T} A x+b^{T} b end{aligned} ∥ A x − b ∥ 2 2 = ( A x − b ) T ( A x − b ) = x T A T A x − 2 b T A x + b T b
根据x x x
∇ x ( x T A T A x − 2 b T A x + b T b ) = ∇ x x T A T A x − ∇ x 2 b T A x + ∇ x b T b = 2 A T A x − 2 A T b egin{aligned}
abla_{x}left(x^{T} A^{T} A x-2 b^{T} A x+b^{T} b
ight) &=
abla_{x} x^{T} A^{T} A x-
abla_{x} 2 b^{T} A x+
abla_{x} b^{T} b \ &=2 A^{T} A x-2 A^{T} b end{aligned} ∇ x ( x T A T A x − 2 b T A x + b T b ) = ∇ x x T A T A x − ∇ x 2 b T A x + ∇ x b T b = 2 A T A x − 2 A T b
将最后一个表达式设置为零，然后解出x x x
x = ( A T A ) − 1 A T b x = (A^TA)^{-1}A^Tb x = ( A T A ) − 1 A T b
这和我们在课堂上得到的相同。
现在让我们考虑一种情况，我们找到一个函数相对于矩阵的梯度，也就是说，对于A ∈ R n × n Ain mathbb{R}^{n imes n} A ∈ R n × n ∇ A ∣ A ∣
abla_{A}|A| ∇ A ∣ A ∣
∣ A ∣ = ∑ i = 1 n ( − 1 ) i + j A i j ∣ A i , j ∣ ( for any j ∈ 1 , … , n ) |A|=sum_{i=1}^{n}(-1)^{i+j} A_{i j}left|A_{ackslash i, ackslash j}
ight| quad( ext { for any } j in 1, ldots, n) ∣ A ∣ = i = 1 ∑ n ( − 1 ) i + j A ij ∣ ∣ A i , j ∣ ∣ ( for any j ∈ 1 , … , n )
所以：
∂ ∂ A k ℓ ∣ A ∣ = ∂ ∂ A k ℓ ∑ i = 1 n ( − 1 ) i + j A i j ∣ A i , j ∣ = ( − 1 ) k + ℓ ∣ A k , ℓ ∣ = ( adj ( A ) ) ℓ k frac{partial}{partial A_{k ell}}|A|=frac{partial}{partial A_{k ell}} sum_{i=1}^{n}(-1)^{i+j} A_{i j}left|A_{ackslash i, ackslash j}
ight|=(-1)^{k+ell}left|A_{ackslash k,ackslash ell}
ight|=(operatorname{adj}(A))_{ell k} ∂ A k ℓ ∂ ∣ A ∣ = ∂ A k ℓ ∂ i = 1 ∑ n ( − 1 ) i + j A ij ∣ ∣ A i , j ∣ ∣ = ( − 1 ) k + ℓ ∣ ∣ A k , ℓ ∣ ∣ = ( adj ( A ) ) ℓ k
从这里可以知道，它直接从伴随矩阵的性质得出：
∇ A ∣ A ∣ = ( adj ( A ) ) T = ∣ A ∣ A − T
abla_{A}|A|=(operatorname{adj}(A))^{T}=|A| A^{-T} ∇ A ∣ A ∣ = ( adj ( A ) ) T = ∣ A ∣ A − T
现在我们来考虑函数f : S + + n → R f : mathbb{S}_{++}^{n}
ightarrow mathbb{R} f : S ++ n → R f ( A ) = log ∣ A ∣ f(A)=log |A| f ( A ) = log ∣ A ∣ f f f ∣ A ∣ > 0 |A|>0 ∣ A ∣ > 0 ∣ A ∣ |A| ∣ A ∣
∂ log ∣ A ∣ ∂ A i j = ∂ log ∣ A ∣ ∂ ∣ A ∣ ∂ ∣ A ∣ ∂ A i j = 1 ∣ A ∣ ∂ ∣ A ∣ ∂ A i j frac{partial log |A|}{partial A_{i j}}=frac{partial log |A|}{partial|A|} frac{partial|A|}{partial A_{i j}}=frac{1}{|A|} frac{partial|A|}{partial A_{i j}} ∂ A ij ∂ log ∣ A ∣ = ∂ ∣ A ∣ ∂ log ∣ A ∣ ∂ A ij ∂ ∣ A ∣ = ∣ A ∣ 1 ∂ A ij ∂ ∣ A ∣
从这一点可以明显看出：
∇ A log ∣ A ∣ = 1 ∣ A ∣ ∇ A ∣ A ∣ = A − 1
abla_{A} log |A|=frac{1}{|A|}
abla_{A}|A|=A^{-1} ∇ A log ∣ A ∣ = ∣ A ∣ 1 ∇ A ∣ A ∣ = A − 1
我们可以在最后一个表达式中删除转置，因为A A A ∂ / ( ∂ x ) log x = 1 / x partial /(partial x) log x=1 / x ∂ / ( ∂ x ) log x = 1/ x
最后，我们使用矩阵演算以直接导致特征值/特征向量分析的方式求解优化问题。 考虑以下等式约束优化问题：
max x ∈ R n x T A x subject to ∥ x ∥ 2 2 = 1 max _{x in mathbb{R}^{n}} x^{T} A x quad ext { subject to }|x|_{2}^{2}=1 x ∈ R n max x T A x subject to ∥ x ∥ 2 2 = 1
对于对称矩阵A ∈ S n Ain mathbb{S}^{n} A ∈ S n 拉格朗日
L ( x , λ ) = x T A x − λ x T x mathcal{L}(x, lambda)=x^{T} A x-lambda x^{T} x L ( x , λ ) = x T A x − λ x T x
其中，λ lambda λ x ⋆ x^{star} x ⋆ x ⋆ x^star x ⋆
∇ x L ( x , λ ) = ∇ x ( x T A x − λ x T x ) = 2 A T x − 2 λ x = 0
abla_{x} mathcal{L}(x, lambda)=
abla_{x}left(x^{T} A x-lambda x^{T} x
ight)=2 A^{T} x-2 lambda x=0 ∇ x L ( x , λ ) = ∇ x ( x T A x − λ x T x ) = 2 A T x − 2 λ x = 0
请注意，这只是线性方程A x = λ x Ax =lambda x A x = λ x x T x = 1 x^T x = 1 x T x = 1 x T A x x^T Ax x T A x A A A
column vector 列向量 row vector 行向量 inner product 内积 dot product 点积 outer product 外积 linear combination 线性组合 identity matrix 单位矩阵 diagonal matrix 对角矩阵 transpose 转置 symmetric matrix 对称矩阵 anti-symmetric matrix 反对称矩阵 trace 迹 norm 范数 (linearly) independent 线性无关 (linearly) dependent 线性相关 column rank 列秩 row rank 行秩 rank 秩 full rank 满秩 inverse 逆 invertible 可逆的 non-singular 非奇异 non-invertible 不可逆 singular 奇异 orthogonal 正交 normalized 归一化 span 张成 projection 投影 range 值域 columnspace 列空间 nullspace 零空间 orthogonal complements 正交补 determinant 行列式 classical adjoint(adjugate) matrix 经典伴随矩阵 adjoint(adjugate) matrix 伴随矩阵 minor 余子式 cofactor 代数余子式 cofactor matrix 代数余子式矩阵 quadratic form 二次型 positive definite(PD) 正定 positive semidefinitee (PSD) 半正定 negative definite (ND) 负定 negative semidefinite(NSD) 半负定 indefinite 不定 Gram matrix 格拉姆矩阵 eigenvalue 特征值 eigenvector 特征向量 Diagonalizing 对角化 gradient 梯度 Hessian 黑塞矩阵 Lagrangian 拉格朗日
翻译不易，https://github.com/openjw/open/blob/master/MachineLearning/cs229/cs229-linalg.md