反向传播算法

当网络给出预测之后，需要根据预测值与实际标签的差异调整网络中的权重和偏置，以便模型在将来能够更好地预测。这个调整过程称为反向传播（误差计算 → 梯度计算 → 参数更新）。

神经网络的结构·

假设一共有 $L$ 层网络，激活函数为 $\sigma$，$z_j^l$ 表示未激活的状态，$a_j^l$ 表示激活后的状态。

$$\begin{aligned} z_j^{(l)} &= \sum_{k = 1}^K{\omega_{jk}^{(l)}a_k^{(l-1)}+b_j^{(l)}}\\ a_j^{(l)} &= \sigma(z_j^{(l)}) \end{aligned}$$

损失函数为

$$C = \frac{1}{2}||y-a^{(L)}||^2 = \frac{1}{2}\sum_{k = 1}^{K}(y_k-a_k^{(L)})^2$$

损失函数对 $a_j^{(L)}$ 的偏导数为

$$\frac{\partial C}{\partial a_j^{(L)}}= a_j^{(L)}-y_j$$

基本方程·

为了实现参数更新，我们需要计算 $\dfrac{\partial C}{\partial \omega_{jk}^{(l)}}$ 和 $\dfrac{\partial C}{\partial b_j^{(l)}}$，

$$\begin{aligned} \dfrac{\partial C}{\partial \omega_{jk}^{(l)}} &=\dfrac{\partial C}{\partial z_j^{(l)}}\dfrac{\partial z_j^{(l)}}{\partial \omega_{jk}^{(l)}}= \dfrac{\partial C}{\partial z_j^{(l)}}a_k^{(l-1)}\\ \dfrac{\partial C}{\partial b_j^{(l)}} &= \dfrac{\partial C}{\partial z_j^{(l)}}\dfrac{\partial z_j^{(l)}}{\partial b_j^{(l)}}= \dfrac{\partial C}{\partial z_j^{(l)}}\cdot 1 \end{aligned}$$

其中涉及到激活函数即 $a_j^{(l)}=\sigma(z_j^{(l)})$，为了简化计算，先定义一个中间变量 $\delta_j^{(l)}$：

$$\delta_j^{(l)} = \frac{\partial C}{\partial z_j^{(l)}}$$

输出层的 $\delta_j^{(L)}$

$$\delta_j^{(L)} = \frac{\partial C}{\partial z_j^{(L)}}=\frac{\partial C}{\partial a_j^{(L)}}\frac{\partial a_j^{(L)}}{\partial z_j^{(L)}} = (a_j^{(L)}-y_j)\sigma'(z_j^{(L)})$$

推广到 $\delta^{(L)}$，得到

$$\begin{aligned} \delta^{(L)} = \begin{bmatrix} \dfrac{\partial C}{\partial a_1^{(L)}}\sigma'(z_1^{(L)})\\ \dfrac{\partial C}{\partial a_2^{(L)}}\sigma'(z_2^{(L)})\\ \cdots\\ \dfrac{\partial C}{\partial a_j^{(L)}}\sigma'(z_j^{(L)})\\ \end{bmatrix} = \nabla_a C \cdot \sigma'(z^{(L)}) \end{aligned}$$

对于 $L-1$ 层，

$$\begin{aligned} \delta_j^{(L-1)} &= \dfrac{\partial C}{\partial z_j^{(L-1)}} =\dfrac{\partial C}{\partial a_j^{(L-1)}}\dfrac{\partial a_j^{(L-1)}}{\partial z_j^{(L-1)}}=\dfrac{\partial C}{\partial a_j^{(L-1)}} \sigma'(z_j^{(L-1)})\\ &= (\sum_{k = 1}^{K}\delta_k^{(L)}\omega_{kj}^{(L)}) \sigma'(z_j^{(L-1)}) \end{aligned}$$

其中 $\dfrac{\partial C}{\partial a_j^{(L-1)}}$ 中 $a_j^{(L-1)}$ 影响了图中红线所示部分

$$\dfrac{\partial C}{\partial a_1^{(L-1)}} =\sum_{k = 1}^{K}\dfrac{\partial C}{\partial z_k^{(L)}}\dfrac{\partial z_k^{(L)}}{\partial a_1^{(L-1)}}=\sum_{k = 1}^{K}\delta_k^{(L)}\omega_{k1}^{(L)}$$

同理，可以得到

$$\dfrac{\partial C}{\partial a_j^{(L-1)}} =\sum_{k = 1}^{K}\dfrac{\partial C}{\partial z_k^{(L)}}\dfrac{\partial z_k^{(L)}}{\partial a_j^{(L-1)}}=\sum_{k = 1}^{K}\delta_k^{(L)}\omega_{kj}^{(L)}$$

对于任意第 $l$ 层，

$$\begin{aligned} \delta_j^{(l)} &= (\sum_{k = 1}^{K}\delta_k^{(l+1)}\omega_{kj}^{(l+1)}) \sigma'(z_j^{(l)})\\ &= ((\omega_j^{(l+1)})^T\delta^{(l+1)}) \sigma'(z_j^{l}) \end{aligned}$$

---

下面计算 $\dfrac{\partial C}{\partial \omega_{jk}^{(l)}}$ 和 $\dfrac{\partial C}{\partial b_j^{(l)}}$，

$$\begin{aligned} \dfrac{\partial C}{\partial \omega_{jk}^{(l)}} &=\dfrac{\partial C}{\partial z_j^{(l)}}\dfrac{\partial z_j^{(l)}}{\partial \omega_{jk}^{(l)}}= \delta_j^{(l)}a_k^{(l-1)}\\ \dfrac{\partial C}{\partial b_j^{(l)}} &= \dfrac{\partial C}{\partial z_j^{(l)}}\dfrac{\partial z_j^{(l)}}{\partial b_j^{(l)}}= \delta_j^{(l)} \end{aligned}$$

算法流程·

1. 输入数据
2. 前向传播

$$\begin{aligned} z_j^{(l)} &= \omega_{j}^{(l)^T}a^{(l-1)}+b_j^{(l)}\\ a_j^{(l)} &= \sigma(z_j^{(l)}) \end{aligned}$$

3. 反向传播误差

$$\begin{aligned} \delta^{(L)} &= \nabla_a C \cdot \sigma'(z^{(L)})\\ \delta_j^{(l)} &= ((\omega_j^{(l+1)})^T\delta^{(l+1)}) \sigma'(z_j^{l}) \end{aligned}$$

4. 梯度下降，更新参数

$$\begin{aligned} \omega_{kj}^{(l)} &= \omega_{kj}^{(l)} - \dfrac{\alpha}{m}\delta_j^{(l)}a_k^{(l-1)}\\ b_j^{(l)} &= b^{(l)} - \dfrac{\alpha}{m}\delta^{(l)} \end{aligned}$$

代码实现·

def backprop(self, x, y):
    """Return a tuple ``(nabla_b, nabla_w)`` representing the
    nablaient for the cost function C_x.  ``nabla_b`` and
    ``nabla_w`` are layer-by-layer lists of numpy arrays, similar
    to ``self.biases`` and ``self.weights``."""
    nabla_b = [np.zeros(b.shape) for b in self.biases]
    nabla_w = [np.zeros(w.shape) for w in self.weights]
    # Forward propagation
    activation = x
    activations = [x] # list to store all the activations, layer by layer
    zs = [] # list to store all the z vectors, layer by layer
    for b, w in zip(self.biases, self.weights):
        z = np.dot(w, activation)+b
        zs.append(z)
        activation = sigmoid(z)
        activations.append(activation)
    #  Backward propagation
    delta = self.cost_derivative(activations[-1], y) * \
        sigmoid_prime(zs[-1])
    nabla_b[-1] = delta
    nabla_w[-1] = np.dot(delta, activations[-2].transpose())
    # Note that the variable l in the loop below is used a little
    # differently to the notation in Chapter 2 of the book.  Here,
    # l = 1 means the last layer of neurons, l = 2 is the
    # second-last layer, and so on.  It's a renumbering of the
    # scheme in the book, used here to take advantage of the fact
    # that Python can use negative indices in lists.
    for l in xrange(2, self.num_layers):
        z = zs[-l]
        sp = sigmoid_prime(z)
        delta = np.dot(self.weights[-l+1].transpose(), delta) * sp
        nabla_b[-l] = delta
        nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())

    return (nabla_b, nabla_w)

参考资料·

1. mnielsen/neural-networks-and-deep-learning