2023-02-21

2022Fall_人工智能笔记

1. 人工智能笔记
1.13. RNN

人工智能笔记

机器学习概览

总结一下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，机器学习善于“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：

需要进行大量手工调整或需要拥有长串规则才能解决的问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：机器学习算法通常可以简化代码“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、提高性能“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
问题复杂“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，传统方法难以解决“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：最好的机器学习方法可以找到解决方案“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
环境有波动“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：机器学习算法可以适应新数据
洞察复杂问题和大量数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

监督学习和非监督学习

在监督学习中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，用来训练算法的训练数据包含了答案“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，称为标签“bd-box<h-char class=“bd bd-beg“>“>（图 1-5“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

图 1-5 用于监督学习“bd-box<h-char class=“bd bd-beg“>“>（比如垃圾邮件分类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）的加了标签的训练集

一个典型的监督学习任务是分类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。垃圾邮件过滤器就是一个很好的例子“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：用许多带有归类“bd-box<h-char class=“bd bd-beg“>“>（垃圾邮件或普通邮件“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）的邮件样本进行训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，过滤器必须还能对新邮件进行分类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

另一个典型任务是预测目标数值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，例如给出一些特征“bd-box<h-char class=“bd bd-beg“>“>（里程数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、车龄“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、品牌等等“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）称作预测值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，来预测一辆汽车的价格“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这类任务称作回归“bd-box<h-char class=“bd bd-beg“>“>（图 1-6“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。要训练这个系统“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你需要给出大量汽车样本“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，包括它们的预测值和标签“bd-box<h-char class=“bd bd-beg“>“>（即“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它们的价格“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

下面是一些重要的监督学习算法“bd-box<h-char class=“bd bd-beg“>“>（本书都有介绍“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>：

K 近邻算法
线性回归
逻辑回归
支持向量机“bd-box<h-char class=“bd bd-beg“>“>（SVM“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）
决策树和随机森林
神经网络

在非监督学习中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可能猜到了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，训练数据是没有加标签的“bd-box<h-char class=“bd bd-beg“>“>（图 1-7“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。系统在没有老师的条件下进行学习“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

下面是一些最重要的非监督学习算法“bd-box<h-char class=“bd bd-beg“>“>（我们会在第 8 章介绍降维“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>：

聚类
K 均值
层次聚类分析“bd-box<h-char class=“bd bd-beg“>“>（Hierarchical Cluster Analysis“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，HCA“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）
期望最大值
可视化和降维
主成分分析“bd-box<h-char class=“bd bd-beg“>“>（Principal Component Analysis“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，PCA“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）
核主成分分析
局部线性嵌入“bd-box<h-char class=“bd bd-beg“>“>（Locally-Linear Embedding“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，LLE“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）
t-分布邻域嵌入算法“bd-box<h-char class=“bd bd-beg“>“>（t-distributed Stochastic Neighbor Embedding“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，t-SNE“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）
关联性规则学习
Apriori 算法
Eclat 算法

一些算法可以处理部分带标签的训练数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，通常是大量不带标签数据加上小部分带标签数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这称作半监督学习“bd-box<h-char class=“bd bd-beg“>“>（图 1-11“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

一些图片存储服务“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，比如 Google Photos“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，是半监督学习的好例子“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。一旦你上传了所有家庭相片“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它就能自动识别到人物 A 出现在了相片 1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、5“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、11 中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，另一个人 B 出现在了相片 2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、5“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、7 中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这是算法的非监督部分“bd-box<h-char class=“bd bd-beg“>“>（聚类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。现在系统需要的就是你告诉它这两个人是谁“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。只要给每个人一个标签“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，算法就可以命名每张照片中的每个人“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，特别适合搜索照片“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

强化学习

强化学习非常不同“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。学习系统在这里被称为智能体“bd-box<h-char class=“bd bd-beg“>“>（agent“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，可以对环境进行观察“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、选择和执行动作“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并获得奖励作为回报“bd-box<h-char class=“bd bd-beg“>“>（负奖励是惩罚“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，见图 1-12“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。然后它必须自己学习哪个是最佳方法“bd-box<h-char class=“bd bd-beg“>“>（称为策略“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，policy“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，以得到长久的最大奖励“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。策略决定了智能体在给定情况下应该采取的行动“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

批量和在线学习

批量学习

在批量学习中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，系统不能进行持续学习“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：必须用所有可用数据进行训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这通常会占用大量时间和计算资源“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以一般是线下做的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。首先是进行训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后部署在生产环境且停止学习“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它只是使用已经学到的策略“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这称为离线学习“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

如果你想让一个批量学习系统明白新数据“bd-box<h-char class=“bd bd-beg“>“>（例如垃圾邮件的新类型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，就需要从头训练一个系统的新版本“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用全部数据集“bd-box<h-char class=“bd bd-beg“>“>（不仅有新数据也有老数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，然后停掉老系统“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，换上新系统“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

幸运的是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、评估“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、部署一套机器学习的系统的整个过程可以自动进行“bd-box<h-char class=“bd bd-beg“>“>（见图 1-3“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，所以即便是批量学习也可以适应改变“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。只要有需要“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，就可以方便地更新数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、训练一个新版本“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

这个方法很简单“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，通常可以满足需求“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是用全部数据集进行训练会花费大量时间“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以一般是每 24 小时或每周训练一个新系统“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果系统需要快速适应变化的数据“bd-box<h-char class=“bd bd-beg“>“>（比如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，预测股价变化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，就需要一个响应更及时的方案“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

在线学习

在在线学习中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，是用数据实例持续地进行训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以一次一个或一次几个实例“bd-box<h-char class=“bd bd-beg“>“>（称为小批量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。每个学习步骤都很快且廉价“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以系统可以动态地学习收到的最新数据“bd-box<h-char class=“bd bd-beg“>“>（见图 1-13“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

图 1-13 在线学习

在线学习很适合系统接收连续流的数据“bd-box<h-char class=“bd bd-beg“>“>（比如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，股票价格“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，且需要自动对改变作出调整“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果计算资源有限“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在线学习是一个不错的方案“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：一旦在线学习系统学习了新的数据实例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它就不再需要这些数据了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以扔掉这些数据“bd-box<h-char class=“bd bd-beg“>“>（除非你想滚回到之前的一个状态“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，再次使用数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这样可以节省大量的空间“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

在线学习算法也适用于在超大数据集“bd-box<h-char class=“bd bd-beg“>“>（一台计算机不足以用于存储它“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）上训练系统“bd-box<h-char class=“bd bd-beg“>“>（这称作核外学习“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，out-of-core learning“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。算法每次只加载部分数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，用这些数据进行训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后重复这个过程“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，直到使用完所有数据“bd-box<h-char class=“bd bd-beg“>“>（见图 1-14“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

警告“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：这个整个过程通常是离线完成的“bd-box<h-char class=“bd bd-beg“>“>（即“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，不在部署的系统上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，所以在线学习这个名字会让人疑惑“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。可以把它想成持续学习“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

图 1-14 使用在线学习处理大量数据集

在线学习系统的一个重要参数是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它们可以多快地适应数据的改变“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：这被称为学习速率“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果你设定一个高学习速率“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，系统就可以快速适应新数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是也会快速忘记老数据“bd-box<h-char class=“bd bd-beg“>“>（你可不想让垃圾邮件过滤器只标记最新的垃圾邮件种类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。相反的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果你设定的学习速率低“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，系统的惰性就会强“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：即“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它学的更慢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但对新数据中的噪声或没有代表性的数据点结果不那么敏感“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

在线学习的挑战之一是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果坏数据被用来进行训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，系统的性能就会逐渐下滑“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果这是一个部署的系统“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，用户就会注意到“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，坏数据可能来自失灵的传感器或机器人“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，或某人向搜索引擎传入垃圾信息以提高搜索排名“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。要减小这种风险“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你需要密集监测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果检测到性能下降“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，要快速关闭“bd-box<h-char class=“bd bd-beg“>“>（或是滚回到一个之前的状态“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。你可能还要监测输入数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，对反常数据做出反应“bd-box<h-char class=“bd bd-beg“>“>（比如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用异常检测算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

线性回归

在第一章“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们介绍了一个简单的生活满意度回归模型:

这个模型仅仅是输入量GDP_per_capita的线性函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，θ[0]和θ[1]是这个模型的参数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，线性模型更一般化的描述指通过计算输入变量的加权和“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并加上一个常数偏置项“bd-box<h-char class=“bd bd-beg“>“>（截距项“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）来得到一个预测值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如公式 4-1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：

公式 4-1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：线性回归预测模型

y_hat表示预测结果
n表示特征的个数
x[i]表示第i个特征的值
θ[j]表示第j个参数“bd-box<h-char class=“bd bd-beg“>“>（包括偏置项θ[0]和特征权重值θ[1], θ[2], ..., θ[nj]“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

上述公式可以写成更为简洁的向量形式“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如公式 4-2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：

公式 4-2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：线性回归预测模型“bd-box<h-char class=“bd bd-beg“>“>（向量形式“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

θ表示模型的参数向量包括偏置项θ[0]和特征权重值θ[1]到θ[n]
θ^T表示向量θ的转置“bd-box<h-char class=“bd bd-beg“>“>（行向量变为了列向量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）
x为每个样本中特征值的向量形式“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，包括x[1]到x[n]“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而且x[0]恒为 1
θ^T · x表示θ^T和x的点积
h[θ]表示参数为θ的假设函数

怎么样去训练一个线性回归模型呢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>？好吧“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，回想一下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，训练一个模型指的是设置模型的参数使得这个模型在训练集的表现较好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

为此“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们首先需要找到一个衡量模型好坏的评定方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在第二章“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们介绍到在回归模型上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最常见的评定标准是均方根误差“bd-box<h-char class=“bd bd-beg“>“>（RMSE“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，详见公式 2-1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。因此“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，为了训练一个线性回归模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你需要找到一个θ值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它使得均方根误差“bd-box<h-char class=“bd bd-beg“>“>（标准误差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）达到最小值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

实践过程中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最小化均方误差比最小化均方根误差更加的简单“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这两个过程会得到相同的θ“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因为函数在最小值时候的自变量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，同样能使函数的方根运算得到最小值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

在训练集X上使用公式 4-3 来计算线性回归假设h[θ]的均方差“bd-box<h-char class=“bd bd-beg“>“>（MSE“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

公式 4-3“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：线性回归模型的 MSE 损失函数

公式中符号的含义大多数都在第二章“bd-box<h-char class=“bd bd-beg“>“>（详见“符号”“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）进行了说明“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，不同的是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：为了突出模型的参数向量θ“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用h[θ]来代替h“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。以后的使用中为了公式的简洁“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用MSE(θ)来代替MSE(X, h[θ])“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

正规方程求解MSE

为了找到最小化损失函数的θ值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以采用公式解“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，换句话说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，就是可以通过解正规方程直接得到最后的结果“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

公式 4-4“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：正规方程

θ_hat指最小化损失θ的值
y是一个向量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，其包含了y^(1)到y^(m)的值

正规方程求解的方法问题在于:

正规方程需要计算矩阵X^T · X的逆“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它是一个n * n的矩阵“bd-box<h-char class=“bd bd-beg“>“>（n是特征的个数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这样一个矩阵求逆的运算复杂度大约在O(n^2.4)到O(n^3)之间“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，具体值取决于计算方式“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。换句话说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果你将你的特征个数翻倍的话“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，其计算时间大概会变为原来的 5.3“bd-box<h-char class=“bd bd-beg“>“>（2^2.4“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）到 8“bd-box<h-char class=“bd bd-beg“>“>（2^3“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）倍“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

提示

当特征的个数较大的时候“bd-box<h-char class=“bd bd-beg“>“>（例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：特征数量为 100000“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，正规方程求解将会非常慢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

有利的一面是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这个方程在训练集上对于每一个实例来说是线性的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，其复杂度为O(m)“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因此只要有能放得下它的内存空间“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它就可以对大规模数据进行训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。同时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一旦你得到了线性回归模型“bd-box<h-char class=“bd bd-beg“>“>（通过解正规方程或者其他的算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，进行预测是非常快的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。因为模型中计算复杂度对于要进行预测的实例数量和特征个数都是线性的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。换句话说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，当实例个数变为原来的两倍多的时候“bd-box<h-char class=“bd bd-beg“>“>（或特征个数变为原来的两倍多“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，预测时间也仅仅是原来的两倍多“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

如何训练模型?梯度下降

梯度下降是一种非常通用的优化算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它能够很好地解决一系列问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。梯度下降的整体思路是通过的迭代来逐渐调整参数使得损失函数达到最小值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

假设浓雾下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你迷失在了大山中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你只能感受到自己脚下的坡度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。为了最快到达山底“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一个最好的方法就是沿着坡度最陡的地方下山“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这其实就是梯度下降所做的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：它计算误差函数关于参数向量Θ的局部梯度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，同时它沿着梯度下降的方向进行下一次迭代“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。当梯度值为零的时候“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，就达到了误差函数最小值 “bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

具体来说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，开始时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，需要选定一个随机的Θ“bd-box<h-char class=“bd bd-beg“>“>（这个值称为随机初始值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，然后逐渐去改进它“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每一次变化一小步“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每一步都试着降低损失函数“bd-box<h-char class=“bd bd-beg“>“>（例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：均方差损失函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，直到算法收敛到一个最小值“bd-box<h-char class=“bd bd-beg“>“>（如图“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：4-3“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

学习率learning rate

在梯度下降中一个重要的参数是步长“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，超参数学习率的值决定了步长的大小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果学习率太小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，必须经过多次迭代“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，算法才能收敛“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这是非常耗时的“bd-box<h-char class=“bd bd-beg“>“>（如图4-4“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

另一方面“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果学习率太大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你将跳过最低点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，到达山谷的另一面“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可能下一次的值比上一次还要大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这可能使的算法是发散的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，函数值变得越来越大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，永远不可能找到一个好的答案“bd-box<h-char class=“bd bd-beg“>“>（如图4-5“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

最后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并不是所有的损失函数看起来都像一个规则的碗“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它们可能是洞“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，山脊“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，高原和各种不规则的地形“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使它们收敛到最小值非常的困难“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。图4-6 显示了梯度下降的两个主要挑战“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：如果随机初始值选在了图像的左侧“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，则它将收敛到局部最小值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这个值要比全局最小值要大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果它从右侧开始“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，那么跨越高原将需要很长时间“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果你早早地结束训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你将永远到不了全局最小值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

事实上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，损失函数的图像呈现碗状“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是不同特征的取值范围相差较大的时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这个碗可能是细长的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。图4-7 展示了梯度下降在不同训练集上的表现“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在左图中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，特征 1 和特征 2 有着相同的数值尺度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在右图中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，特征 1 比特征 2 的取值要小的多“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，由于特征 1 较小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因此损失函数改变时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，Θ[1]会有较大的变化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，于是这个图像会在Θ[1]轴方向变得细长“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

当我们使用梯度下降的时候“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，应该确保所有的特征有着相近的尺度范围“bd-box<h-char class=“bd bd-beg“>“>（例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：使用 Scikit Learn 的 StandardScaler类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，否则它将需要很长的时间才能够收敛“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

批量梯度下降

使用梯度下降的过程中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你需要计算每一个Θ[j]下损失函数的梯度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。换句话说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你需要计算当Θ[j]变化一点点时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，损失函数改变了多少“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这称为偏导数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它就像当你面对东方的时候问“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：<h-char class=“bd bd-beg“>“我脚下的坡度是多少“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>？<h-char class=“bd bd-beg“>“<h-char class=“bd bd-beg“>。然后面向北方的时候问同样的问题“bd-box<h-char class=“bd bd-beg“>“>（如果你能想象一个超过三维的宇宙“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以对所有的方向都这样做“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。公式 4-5 计算关于Θ[j]的损失函数的偏导数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，记为“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：∂MSE/∂θ[j]“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

为了避免单独计算每一个梯度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你也可以使用公式 4-6 来一起计算它们“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。梯度向量记为ᐁ[θ]MSE(θ)“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，其包含了损失函数所有的偏导数“bd-box<h-char class=“bd bd-beg“>“>（每个模型参数只出现一次“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

随机梯度下降

批量梯度下降的最要问题是计算每一步的梯度时都需要使用整个训练集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这导致在规模较大的数据集上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，其会变得非常的慢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。与其完全相反的随机梯度下降“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在每一步的梯度计算上只随机选取训练集中的一个样本“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。很明显“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，由于每一次的操作都使用了非常少的数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这样使得算法变得非常快“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。由于每一次迭代“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，只需要在内存中有一个实例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这使随机梯度算法可以在大规模训练集上使用“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

虽然随机性可以很好的跳过局部最优值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但同时它却不能达到最小值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。解决这个难题的一个办法是逐渐降低学习率“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。开始时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，走的每一步较大“bd-box<h-char class=“bd bd-beg“>“>（这有助于快速前进同时跳过局部最小值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，然后变得越来越小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，从而使算法到达全局最小值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这个过程被称为模拟退火“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因为它类似于熔融金属慢慢冷却的冶金学退火过程“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。决定每次迭代的学习率的函数称为learning schedule“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果学习速度降低得过快“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可能会陷入局部最小值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，甚至在到达最小值的半路就停止了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果学习速度降低得太慢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可能在最小值的附近长时间摆动“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，同时如果过早停止训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最终只会出现次优解“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

由于每个实例的选择是随机的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，有的实例可能在每一代中都被选到“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这样其他的实例也可能一直不被选到“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果你想保证每一代迭代过程“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，算法可以遍历所有实例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一种方法是将训练集打乱重排“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后选择一个实例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，之后再继续打乱重排“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，以此类推一直进行下去“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。但是这样收敛速度会非常的慢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

小批量梯度下降

最后一个梯度下降算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们将介绍小批量梯度下降算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。一旦你理解了批量梯度下降和随机梯度下降“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，再去理解小批量梯度下降是非常简单的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在迭代的每一步“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，批量梯度使用整个训练集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，随机梯度时候用仅仅一个实例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在小批量梯度下降中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它则使用一个随机的小型实例集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它比随机梯度的主要优点在于你可以通过矩阵运算的硬件优化得到一个较好的训练表现“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，尤其当你使用 GPU 进行运算的时候“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

学习曲线/过拟合/欠拟合

如果你使用一个高阶的多项式回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可能发现它的拟合程度要比普通的线性回归要好的多“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，图4-14 使用一个 300 阶的多项式模型去拟合之前的数据集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并同简单线性回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、2 阶的多项式回归进行比较“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。注意 300 阶的多项式模型如何摆动以尽可能接近训练实例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

当然“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这种高阶多项式回归模型在这个训练集上严重过拟合了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，线性模型则欠拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在这个训练集上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，二次模型有着较好的泛化能力“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。那是因为在生成数据时使用了二次模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是一般我们不知道这个数据生成函数是什么“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，那我们该如何决定我们模型的复杂度呢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>？你如何告诉我你的模型是过拟合还是欠拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>？

如果一个模型在训练集上表现良好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，通过交叉验证指标却得出其泛化能力很差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，那么你的模型就是过拟合了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果在这两方面都表现不好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，那么它就是欠拟合了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这种方法可以告诉我们“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你的模型是太复杂还是太简单了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
画出模型在训练集上的表现“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，同时画出以训练集规模为自变量的训练集函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。为了得到图像“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，需要在训练集的不同规模子集上进行多次训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。下面的代码定义了一个函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，用来画出给定训练集后的模型学习曲线

上面的曲线表现了一个典型的欠拟合模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，两条曲线都到达高原地带并趋于稳定“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并且最后两条曲线非常接近“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，同时误差值非常大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

这幅图值得我们深究“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。首先“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们观察训练集的表现“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：当训练集只有一两个样本的时候“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，模型能够非常好的拟合它们“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这也是为什么曲线是从零开始的原因“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。但是当加入了一些新的样本的时候“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，训练集上的拟合程度变得难以接受“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，出现这种情况有两个原因“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一是因为数据中含有噪声“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，另一个是数据根本不是线性的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。因此随着数据规模的增大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，误差也会一直增大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，直到达到高原地带并趋于稳定“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在之后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，继续加入新的样本“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，模型的平均误差不会变得更好或者更差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。我们继续来看模型在验证集上的表现“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，当以非常少的样本去训练时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，模型不能恰当的泛化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，也就是为什么验证误差一开始是非常大的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。当训练样本变多的到时候“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，模型学习的东西变多“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，验证误差开始缓慢的下降“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。但是一条直线不可能很好的拟合这些数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因此最后误差会到达在一个高原地带并趋于稳定“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最后和训练集的曲线非常接近“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

这幅图像和之前的有一点点像“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是其有两个非常重要的不同点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：

在训练集上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，误差要比线性回归模型低的多“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
图中的两条曲线之间有间隔“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这意味模型在训练集上的表现要比验证集上好的多“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这也是模型过拟合的显著特点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。当然“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果你使用了更大的训练数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这两条曲线最后会非常的接近“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

误差来源分析

改善模型过拟合的一种方法是提供更多的训练数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，直到训练误差和验证误差相等“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
在统计和机器学习领域有个重要的理论“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：一个模型的泛化误差由三个不同误差的和决定“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：

偏差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：泛化误差的这部分误差是由于错误的假设决定的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如实际是一个二次模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你却假设了一个线性模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。一个高偏差的模型最容易出现欠拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
方差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：这部分误差是由于模型对训练数据的微小变化较为敏感“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一个多自由度的模型更容易有高的方差“bd-box<h-char class=“bd bd-beg“>“>（例如一个高阶多项式模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，因此会导致模型过拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
不可约误差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：这部分误差是由于数据本身的噪声决定的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。降低这部分误差的唯一方法就是进行数据清洗“bd-box<h-char class=“bd bd-beg“>“>（例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：修复数据源“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，修复坏的传感器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，识别和剔除异常值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

正则化

降低模型的过拟合的好方法是正则化这个模型“bd-box<h-char class=“bd bd-beg“>“>（即限制它“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>：模型有越少的自由度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，就越难以拟合数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，正则化一个多项式模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一个简单的方法就是减少多项式的阶数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

对于一个线性模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，正则化的典型实现就是约束模型中参数的权重“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。接下来我们将介绍三种不同约束权重的方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：Ridge 回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，Lasso 回归和 Elastic Net“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

岭回归

岭回归“bd-box<h-char class=“bd bd-beg“>“>（也称为 Tikhonov 正则化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）是线性回归的正则化版“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：在损失函数上直接加上一个正则项α Σ θ[i]^2, i = 1 -> n“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这使得学习算法不仅能够拟合数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而且能够使模型的参数权重尽量的小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。注意到这个正则项只有在训练过程中才会被加到损失函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。当得到完成训练的模型后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们应该使用没有正则化的测量方法去评价模型的表现“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

训练过程使用的损失函数和测试过程使用的评价函数是不一样的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。除了正则化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，还有一个不同“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：训练时的损失函数应该在优化过程中易于求导

岭回归损失函数:

超参数α决定了你想正则化这个模型的强度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果α = 0那此时的岭回归便变为了线性回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果α非常的大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所有的权重最后都接近于零“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最后结果将是一条穿过数据平均值的水平直线“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。对于梯度下降来说仅仅在均方差梯度向量“bd-box<h-char class=“bd bd-beg“>“>（公式 4-6“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）加上一项αw“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

图4-17 展示了在相同线性数据上使用不同α值的岭回归模型最后的表现“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。左图中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用简单的岭回归模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最后得到了线性的预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。右图中的数据首先使用 10 阶的PolynomialFearures进行扩展“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后使用StandardScaler进行缩放“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最后将岭模型应用在处理过后的特征上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这就是带有岭正则项的多项式回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。注意当α增大的时候“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，导致预测曲线变得扁平“bd-box<h-char class=“bd bd-beg“>“>（即少了极端值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，多了一般值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，这样减少了模型的方差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，却增加了模型的偏差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

Lasso回归

Lasso 回归“bd-box<h-char class=“bd bd-beg“>“>（也称 Least Absolute Shrinkage“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，或者 Selection Operator Regression“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）是另一种正则化版的线性回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：就像岭回归那样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它也在损失函数上添加了一个正则化项“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是它使用权重向量的l1范数而不是权重向量l2范数平方的一半“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。（如公式 4-10“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

Lasso 回归的损失函数:

图4-18 展示了和图4-17 相同的事情“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在相同线性数据上使用不同α值的岭回归模型最后的表现“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。左图中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用简单的岭回归模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最后得到了线性的预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。右图中的数据首先使用 10 阶的PolynomialFearures进行扩展“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后使用StandardScaler进行缩放“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最后用 Lasso 模型代替了 Ridge 模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，同时调小了α的值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

Lasso回归的一个重要特征是它倾向于完全消除最不重要的特征的权重“bd-box<h-char class=“bd bd-beg“>“>（即将它们设置为零“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，右图中的虚线所示“bd-box<h-char class=“bd bd-beg“>“>（α = 10^(-7)“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，曲线看起来像一条二次曲线“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而且几乎是线性的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这是因为所有的高阶多项特征都被设置为零“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。换句话说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，Lasso 回归自动的进行特征选择同时输出一个稀疏模型

弹性网络“bd-box<h-char class=“bd bd-beg“>“>（ElasticNet“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

弹性网络介于 Ridge 回归和 Lasso 回归之间“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它的正则项是 Ridge 回归和 Lasso 回归正则项的简单混合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，同时你可以控制它们的混合率r“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，当r = 0时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，弹性网络就是 Ridge 回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，当r = 1时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，其就是 Lasso 回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。具体表示如公式 4-12“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

弹性网络损失函数

那么我们该如何选择线性回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，岭回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，Lasso 回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，弹性网络呢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>？一般来说有一点正则项的表现更好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因此通常你应该避免使用简单的线性回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。岭回归是一个很好的首选项“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是如果你的特征仅有少数是真正有用的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你应该选择 Lasso 和弹性网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。就像我们讨论的那样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它两能够将无用特征的权重降为零“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。一般来说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，弹性网络的表现要比 Lasso 好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因为当特征数量比样本的数量大的时候“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，或者特征之间有很强的相关性时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，Lasso 可能会表现的不规律“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

_EarlyStopping

对于迭代学习算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，有一种非常特殊的正则化方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，就像梯度下降在验证错误达到最小值时立即停止训练那样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。我们称为早期停止法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。图4-20 表示使用批量梯度下降来训练一个非常复杂的模型“bd-box<h-char class=“bd bd-beg“>“>（一个高阶多项式回归模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。随着训练的进行“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，算法一直学习“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它在训练集上的预测误差“bd-box<h-char class=“bd bd-beg“>“>（RMSE“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）自然而然的下降“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。然而一段时间后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，验证误差停止下降“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并开始上升“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这意味着模型在训练集上开始出现过拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。一旦验证错误达到最小值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，便提早停止训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

机器学习基础

支持向量机SVM

线性SVM分类

支持向量机“bd-box<h-char class=“bd bd-beg“>“>（SVM“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）是个非常强大并且有多种功能的机器学习模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，能够做线性或者非线性的分类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，甚至异常值检测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。机器学习领域中最为流行的模型之一“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，是任何学习机器学习的人必备的工具“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。SVM 特别适合应用于复杂但中小规模数据集的分类问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

SVM 的基本思想能够用一些图片来解释得很好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，图 5-1 展示了我们在第 4 章结尾处介绍的鸢尾花数据集的一部分这两个种类能够被非常清晰“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，非常容易的用一条直线分开“bd-box<h-char class=“bd bd-beg“>“>（即线性可分的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。左边的图显示了三种可能的线性分类器的判定边界“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。其中用虚线表示的线性模型判定边界很差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，甚至不能正确地划分类别“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。另外两个线性模型在这个数据集表现的很好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是它们的判定边界很靠近样本点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在新的数据上可能不会表现的很好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。相比之下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，右边图中 SVM 分类器的判定边界实线“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，不仅分开了两种类别“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而且还尽可能地远离了最靠近的训练数据点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。你可以认为 SVM 分类器在两种类别之间保持了一条尽可能宽敞的街道“bd-box<h-char class=“bd bd-beg“>“>（图中平行的虚线“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，其被称为最大间隔分类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。注意到添加更多的样本点在“街道”外并不会影响到判定边界“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因为判定边界是由位于“街道”边缘的样本点确定的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这些样本点被称为“支持向量”

SVM 对特征缩放比较敏感“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以看到图 5-2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：左边的图中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，垂直的比例要更大于水平的比例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以最宽的“街道”接近水平“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。但对特征缩放后“bd-box<h-char class=“bd bd-beg“>“>（例如使用 Scikit-Learn 的 StandardScaler“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，判定边界看起来要好得多“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如右图“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

软间隔分类

如果我们严格地规定所有的数据都不在“街道”上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，都在正确地两边“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，称为硬间隔分类.
硬间隔分类有两个问题

只对线性可分的数据起作用“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，
对异常点敏感“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
图 5-3 显示了只有一个异常点的鸢尾花数据集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：左边的图中很难找到硬间隔“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，右边的图中判定边界和我们之前在图 5-1 中没有异常点的判定边界非常不一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它很难一般化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

为了避免上述的问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们更倾向于使用更加软性的模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。目的在保持“街道”尽可能大和避免间隔违规“bd-box<h-char class=“bd bd-beg“>“>（例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：数据点出现在“街道”中央或者甚至在错误的一边“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）之间找到一个良好的平衡“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这就是软间隔分类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

在 Scikit-Learn 库的 SVM 类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可以用C超参数“bd-box<h-char class=“bd bd-beg“>“>（惩罚系数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）来控制这种平衡“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：较小的C会导致更宽的“街道”“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但更多的间隔违规“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。图 5-4 显示了在非线性可分隔的数据集上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，两个软间隔 SVM 分类器的判定边界“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。左边图中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用了较大的C值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，导致更少的间隔违规“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是间隔较小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。右边的图“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用了较小的C值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，间隔变大了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是许多数据点出现在了“街道”上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。然而“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，第二个分类器似乎泛化地更好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：事实上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在这个训练数据集上减少了预测错误“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因为实际上大部分的间隔违规点出现在了判定边界正确的一侧“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

如果你的 SVM 模型过拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可以尝试通过减小超参数C去调整“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

非线性SVM分类

尽管线性 SVM 分类器在许多案例上表现得出乎意料的好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是很多数据集并不是线性可分的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。一种处理非线性数据集方法是增加更多的特征“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，例如多项式特征“bd-box<h-char class=“bd bd-beg“>“>（正如你在第 4 章所做的那样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>；在某些情况下可以变成线性可分的数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在图 5-5 的左图中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它只有一个特征x1的简单的数据集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，正如你看到的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，该数据集不是线性可分的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。但是如果你增加了第二个特征 x2=(x1)^2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，产生的 2D 数据集就能很好的线性可分“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

为了实施这个想法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，通过 Scikit-Learn“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可以创建一个流水线“bd-box<h-char class=“bd bd-beg“>“>（Pipeline“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）去包含多项式特征“bd-box<h-char class=“bd bd-beg“>“>（PolynomialFeatures“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）变换“bd-box<h-char class=“bd bd-beg“>“>（在 121 页的“Polynomial Regression”中讨论“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，然后一个StandardScaler和LinearSVC“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.datasets "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import make_moons
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.pipeline "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import Pipeline
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.preprocessing "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import PolynomialFeatures
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">polynomial_svm_clf = Pipeline((
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">        ("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"poly_features", PolynomialFeatures(degree="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">3)),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">        ("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"scaler", StandardScaler()),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">        ("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"svm_clf", LinearSVC(C="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">10, loss="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"hinge"))
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    ))
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">polynomial_svm_clf.fit(X, y)

多项式核

添加多项式特征很容易实现“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，不仅仅在 SVM“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在各种机器学习算法都有不错的表现“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是低次数的多项式不能处理非常复杂的数据集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而高次数的多项式却产生了大量的特征“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，会使模型变得慢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

幸运的是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，当你使用 SVM 时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可以运用一个被称为“核技巧”“bd-box<h-char class=“bd bd-beg“>“>（kernel trick“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）的神奇数学技巧“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它可以取得就像你添加了许多多项式“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，甚至有高次数的多项式“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一样好的结果“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。所以不会大量特征导致的组合爆炸“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因为你并没有增加任何特征“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.svm "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import SVC
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">poly_kernel_svm_clf = Pipeline((
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">        ("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"scaler", StandardScaler()),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">        ("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"svm_clf", SVC(kernel="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"poly", degree="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">3, coef0="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1, C="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">5))
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    ))
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">poly_kernel_svm_clf.fit(X, y)

3 阶的多项式核训练了一个 SVM 分类器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，即图 5-7 的左图“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。右图是使用了 10 阶的多项式核 SVM 分类器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。很明显“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果你的模型过拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可以减小多项式核的阶数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。相反的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果是欠拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可以尝试增大它“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。超参数coef0控制了高阶多项式与低阶多项式对模型的影响“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

通用的方法是用网格搜索“bd-box<h-char class=“bd bd-beg“>“>（grid search 见第 2 章“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）去找到最优超参数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。首先进行非常粗略的网格搜索一般会很快“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后在找到的最佳值进行更细的网格搜索“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

增加相似特征

另一种解决非线性问题的方法是使用相似函数“bd-box<h-char class=“bd bd-beg“>“>（similarity funtion“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）计算每个样本与特定地标“bd-box<h-char class=“bd bd-beg“>“>（landmark“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）的相似度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，让我们来看看前面讨论过的一维数据集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并在x1=-2和x1=1之间增加两个地标“bd-box<h-char class=“bd bd-beg“>“>（图 5-8 左图“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。接下来“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们定义一个相似函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，即高斯径向基函数“bd-box<h-char class=“bd bd-beg“>“>（Gaussian Radial Basis Function“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，RBF“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，设置γ = 0.3“bd-box<h-char class=“bd bd-beg“>“>（见公式 5-1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

你可能想知道如何选择地标“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。最简单的方法是在数据集中的每一个样本的位置创建地标“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这将产生更多的维度从而增加了转换后数据集是线性可分的可能性“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。但缺点是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，m个样本“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，n个特征的训练集被转换成了m个实例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，m个特征的训练集“bd-box<h-char class=“bd bd-beg“>“>（假设你删除了原始特征“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这样一来“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果你的训练集非常大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你最终会得到同样大的特征“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

高斯 RBF 核

就像多项式特征法一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，相似特征法对各种机器学习算法同样也有不错的表现“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。但是在所有额外特征上的计算成本可能很高“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，特别是在大规模的训练集上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。然而“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，“核” 技巧再一次显现了它在 SVM 上的神奇之处“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：高斯核让你可以获得同样好的结果成为可能“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，就像你在相似特征法添加了许多相似特征一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但事实上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你并不需要在 RBF 添加它们“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。我们使用 SVC 类的高斯 RBF 核来检验一下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">rbf_kernel_svm_clf = Pipeline((
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">        ("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"scaler", StandardScaler()),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">        ("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"svm_clf", SVC(kernel="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"rbf", gamma="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">5, C="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.001))
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    ))
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">rbf_kernel_svm_clf.fit(X, y)

这个模型在图 5-9 的左下角表示“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。其他的图显示了用不同的超参数gamma (γ)和C训练的模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。增大γ使钟型曲线更窄“bd-box<h-char class=“bd bd-beg“>“>（图 5-8 左图“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，导致每个样本的影响范围变得更小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：即判定边界最终变得更不规则“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在单个样本周围环绕“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。相反的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，较小的γ值使钟型曲线更宽“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，样本有更大的影响范围“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，判定边界最终则更加平滑“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。所以γ是可调整的超参数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：如果你的模型过拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你应该减小γ值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，若欠拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，则增大γ“bd-box<h-char class=“bd bd-beg“>“>（与超参数C相似“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

计算复杂度

LinearSVC类基于liblinear库“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它实现了线性 SVM 的优化算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它并不支持核技巧“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是它样本和特征的数量几乎是线性的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：训练时间复杂度大约为O(m × n)“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

如果你要非常高的精度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这个算法需要花费更多时间“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这是由容差值超参数ϵ“bd-box<h-char class=“bd bd-beg“>“>（在 Scikit-learn 称为tol“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）控制的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。大多数分类任务中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用默认容差值的效果是已经可以满足一般要求“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

SVC 类基于libsvm库“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它实现了支持核技巧的算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。训练时间复杂度通常介于 O(m^2 × n)和O(m^3 × n)之间. 不幸的是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这意味着当训练样本变大时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它将变得极其慢“bd-box<h-char class=“bd bd-beg“>“>（例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，成千上万个样本“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这个算法对于复杂但小型或中等数量的数据集表现是完美的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。然而“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它能对特征数量很好的缩放“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，尤其对稀疏特征来说“bd-box<h-char class=“bd bd-beg“>“>（sparse features“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）（即每个样本都有一些非零特征“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。在这个情况下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，算法对每个样本的非零特征的平均数量进行大概的缩放“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

表 5-1 对 Scikit-learn 的 SVM 分类模型进行比较“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

SVM回归

SVM 算法应用广泛“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：不仅仅支持线性和非线性的分类任务“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，还支持线性和非线性的回归任务“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。SVM回归核心在于逆转我们的目标“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：**限制间隔违规的情况下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，不是试图在两个类别之间找到尽可能大的“街道”“bd-box<h-char class=“bd bd-beg“>“>（即间隔“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。SVM 回归任务是限制间隔违规情况下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，尽量放置更多的样本在“街道”上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。**“街道”的宽度由超参数ϵ控制“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。图 5-10 显示了在一些随机生成的线性数据上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，两个线性 SVM 回归模型的训练情况“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。一个有较大的间隔“bd-box<h-char class=“bd bd-beg“>“>（ϵ=1.5“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，另一个间隔较小“bd-box<h-char class=“bd bd-beg“>“>（ϵ=0.5“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

你可以使用 Scikit-Learn 的LinearSVR类去实现线性 SVM 回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。下面的代码产生的模型在图 5-10 左图“bd-box<h-char class=“bd bd-beg“>“>（训练数据需要被中心化和标准化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.svm "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import LinearSVR
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">svm_reg = LinearSVR(epsilon="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1.5)
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">svm_reg.fit(X, y)

处理非线性回归任务“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可以使用核化的 SVM 模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。比如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，图 5-11 显示了在随机二次方的训练集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用二次方多项式核函数的 SVM 回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。左图是较小的正则化“bd-box<h-char class=“bd bd-beg“>“>（即更大的C值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，右图则是更大的正则化“bd-box<h-char class=“bd bd-beg“>“>（即小的C值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.svm "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import SVR
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">svm_poly_reg = SVR(kernel="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"poly", degree="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">2, C="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">100, epsilon="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.1)
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">svm_poly_reg.fit(X, y)

决策树

和支持向量机一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，决策树是一种多功能机器学习算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，即可以执行分类任务也可以执行回归任务“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，甚至包括多输出“bd-box<h-char class=“bd bd-beg“>“>（multioutput“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）任务.它是一种功能很强大的算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以对很复杂的数据集进行拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在第二章中我们对加利福尼亚住房数据集使用决策树回归模型进行训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，就很好的拟合了数据集“bd-box<h-char class=“bd bd-beg“>“>（实际上是过拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。决策树也是随机森林的基本组成部分“bd-box<h-char class=“bd bd-beg“>“>（见第 7 章“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，而随机森林是当今最强大的机器学习算法之一“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

分类

理解决策树“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们需要先构建一个决策树并亲身体验它到底如何进行预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。接下来的代码就是在我们熟知的鸢尾花数据集上进行一个决策树分类器的训练

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.datasets "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import load_iris
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.tree "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import DecisionTreeClassifier
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">iris = load_iris()
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">X = iris.data[:, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">2:] "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"comment"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"># petal length and width 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">y = iris.target
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">tree_clf = DecisionTreeClassifier(max_depth="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">2)
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">tree_clf.fit(X, y)

我们的第一个决策树如图 6-1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

决策树的众多特性之一就是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它不需要太多的数据预处理“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，尤其是不需要进行特征的缩放或者归一化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

现在让我们来看看在图 6-1 中的树是如何进行预测的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。假设你找到了一朵鸢尾花并且想对它进行分类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你从根节点开始“bd-box<h-char class=“bd bd-beg“>“>（深度为 0“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，顶部“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>：该节点询问花朵的花瓣长度是否小于 2.45 厘米“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，您将向下移动到根的左侧子节点“bd-box<h-char class=“bd bd-beg“>“>（深度为 1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，左侧“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。在这种情况下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它是一片叶子节点“bd-box<h-char class=“bd bd-beg“>“>（即它没有任何子节点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，所以它不会问任何问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：你可以方便地查看该节点的预测类别“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，决策树预测你的花是 Iris-Setosa“bd-box<h-char class=“bd bd-beg“>“>（class = setosa“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

现在假设你找到了另一朵花“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但这次的花瓣长度是大于 2.45 厘米的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。你必须向下移动到根的右侧子节点“bd-box<h-char class=“bd bd-beg“>“>（深度为 1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，右侧“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，而这个节点不是叶节点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以它会问另一个问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：花瓣宽度是否小于 1.75 厘米“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>？如果是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，那么你的花很可能是一个 Iris-Versicolor“bd-box<h-char class=“bd bd-beg“>“>（深度为 2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，左“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。如果不是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，那很可能一个 Iris-Virginica“bd-box<h-char class=“bd bd-beg“>“>（深度为 2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，右“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

图 6-2 显示了决策树的决策边界“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。粗的垂直线代表根节点“bd-box<h-char class=“bd bd-beg“>“>（深度为 0“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）的决定边界“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：花瓣长度为 2.45 厘米“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。由于左侧区域是纯的“bd-box<h-char class=“bd bd-beg“>“>（只有 Iris-Setosa“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，所以不能再进一步分裂“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。然而“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，右边的区域是不纯的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以深度为 1 的右边节点在花瓣宽度为 1.75 厘米处分裂“bd-box<h-char class=“bd bd-beg“>“>（用虚线表示“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。又由于max_depth设置为 2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，决策树在那里停了下来“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。但是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果将max_depth设置为 3“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，两个深度为 2 的节点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每个都将会添加另一个决策边界“bd-box<h-char class=“bd bd-beg“>“>（用虚线表示“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

决策树非常直观“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，他们的决定很容易被解释“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这种模型通常被称为白盒模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。相反“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，随机森林或神经网络通常被认为是黑盒模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。他们能做出很好的预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并且您可以轻松检查它们做出这些预测过程中计算的执行过程“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。然而“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，人们通常很难用简单的术语来解释为什么模型会做出这样的预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

_CART训练算法

Scikit-Learn 用分裂回归树“bd-box<h-char class=“bd bd-beg“>“>（Classification And Regression Tree“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，简称 CART“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）算法训练决策树“bd-box<h-char class=“bd bd-beg“>“>（也叫“增长树”“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这种算法思想真的非常简单“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：首先使用单个特征k和阈值t[k]“bd-box<h-char class=“bd bd-beg“>“>（例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，“花瓣长度≤2.45cm”“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）将训练集分成两个子集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它如何选择k和t[k]呢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>？它寻找到能够产生最纯粹的子集一对(k, t[k])“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后通过子集大小加权计算“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。算法会尝试最小化成本函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

当它成功的将训练集分成两部分之后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它将会继续使用相同的递归式逻辑继续的分割子集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后是子集的子集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。当达到预定的最大深度之后将会停止分裂“bd-box<h-char class=“bd bd-beg“>“>（由max_depth超参数决定“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，或者是它找不到可以继续降低不纯度的分裂方法的时候“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。几个其他超参数“bd-box<h-char class=“bd bd-beg“>“>（min_samples_split“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，min_samples_leaf“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，min_weight_fraction_leaf“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，max_leaf_nodes“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。控制了其他的停止生长条件

找到最优树是一个 NP 完全问题“bd-box<h-char class=“bd bd-beg“>“>（自行百度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>：它需要O(exp^m)时间“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，即使对于相当小的训练集也会使问题变得棘手“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这就是为什么我们必须设置一个“合理的”“bd-box<h-char class=“bd bd-beg“>“>（而不是最佳的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）解决方案“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

在建立好决策树模型后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，做出预测需要遍历决策树“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，从根节点一直到叶节点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。决策树通常近似左右平衡“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因此遍历决策树需要经历大致O(log2(m)) 个节点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。由于每个节点只需要检查一个特征的值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因此总体预测复杂度仅为O(log2(m))“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，与特征的数量无关“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。所以即使在处理大型训练集时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，预测速度也非常快“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

然而“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，训练算法的时候“bd-box<h-char class=“bd bd-beg“>“>（训练和预测不同“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）需要比较所有特征“bd-box<h-char class=“bd bd-beg“>“>（如果设置了max_features会更少一些“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

在每个节点的所有样本上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。就有了O(n×m log(m))的训练复杂度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。对于小型训练集“bd-box<h-char class=“bd bd-beg“>“>（少于几千例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，Scikit-Learn 可以通过预先设置数据“bd-box<h-char class=“bd bd-beg“>“>（presort = True“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）来加速训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是这对于较大训练集来说会显着减慢训练速度

正则化

如果不添加约束“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，树结构模型通常将根据训练数据调整自己“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使自身能够很好的拟合数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而这种情况下大多数会导致模型过拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

DecisionTreeClassifier类还有一些其他的参数用于限制树模型的形状:

min_samples_split“bd-box<h-char class=“bd bd-beg“>“>（节点在被分裂之前必须具有的最小样本数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，min_samples_leaf“bd-box<h-char class=“bd bd-beg“>“>（叶节点必须具有的最小样本数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，min_weight_fraction_leaf“bd-box<h-char class=“bd bd-beg“>“>（和min_samples_leaf相同“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但表示为加权总数的一小部分实例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，max_leaf_nodes“bd-box<h-char class=“bd bd-beg“>“>（叶节点的最大数量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）和 max_features“bd-box<h-char class=“bd bd-beg“>“>（在每个节点被评估是否分裂的时候“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，具有的最大特征数量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。增加min_* hyperparameters或者减少max_* hyperparameters会使模型正则化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

一些其他算法的工作原理是在没有任何约束条件下训练决策树模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，让模型自由生长“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后再对不需要的节点进行剪枝“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

图 6-3 显示了对moons数据集“bd-box<h-char class=“bd bd-beg“>“>（在第 5 章介绍过“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）进行训练生成的两个决策树模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，左侧的图形对应的决策树使用默认超参数生成“bd-box<h-char class=“bd bd-beg“>“>（没有限制生长条件“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，右边的决策树模型设置为min_samples_leaf=4“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。很明显“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，左边的模型过拟合了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而右边的模型泛用性更好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

回归

决策树也能够执行回归任务“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，让我们使用 Scikit-Learn 的DecisionTreeRegressor类构建一个回归树“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，让我们用max_depth = 2在具有噪声的二次项数据集上进行训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.tree "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import DecisionTreeRegressor
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">tree_reg = DecisionTreeRegressor(max_depth="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">2)
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">tree_reg.fit(X, y)

这棵树看起来非常类似于你之前建立的分类树“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它的主要区别在于“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它不是预测每个节点中的样本所属的分类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而是预测一个具体的数值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，假设您想对x[1] = 0.6的新实例进行预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。从根开始遍历树“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最终到达预测值等于 0.1106 的叶节点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。该预测仅仅是与该叶节点相关的 110 个训练实例的平均目标值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。而这个预测结果在对应的 110 个实例上的均方误差“bd-box<h-char class=“bd bd-beg“>“>（MSE“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）等于 0.0151“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

CART 算法的工作方式与之前处理分类模型基本一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，不同之处在于“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，现在不再以最小化不纯度的方式分割训练集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而是试图以最小化 MSE 的方式分割训练集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。公式 6-4 显示了成本函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，该算法试图最小化这个成本函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

和处理分类任务时一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，决策树在处理回归问题的时候也容易过拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果不添加任何正则化“bd-box<h-char class=“bd bd-beg“>“>（默认的超参数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，你就会得到图 6-6 左侧的预测结果“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，显然“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，过度拟合的程度非常严重“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。而当我们设置了min_samples_leaf = 10“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，相对就会产生一个更加合适的模型了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，就如图 6-6 所示的那样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

不稳定性

它很容易理解和解释“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，易于使用且功能丰富而强大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。然而“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它也有一些限制“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，首先“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可能已经注意到了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，==决策树很喜欢设定正交化的决策边界“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，==“bd-box<h-char class=“bd bd-beg“>“>（所有边界都是和某一个轴相垂直的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，这使得它对训练数据集的旋转很敏感“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，例如图 6-7 显示了一个简单的线性可分数据集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在左图中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，决策树可以轻易的将数据分隔开“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是在右图中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，当我们把数据旋转了 45° 之后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，决策树的边界看起来变的格外复杂“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。尽管两个决策树都完美的拟合了训练数据“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，右边模型的泛化能力很可能非常差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

解决这个难题的一种方式是使用 PCA 主成分分析“bd-box<h-char class=“bd bd-beg“>“>（第八章“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，这样通常能使训练结果变得更好一些“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

更加通俗的讲“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，==决策时的主要问题是它对训练数据的微小变化非常敏感“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，==举例来说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们仅仅从鸢尾花训练数据中将最宽的 Iris-Versicolor 拿掉“bd-box<h-char class=“bd bd-beg“>“>（花瓣长 4.8 厘米“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，宽 1.8 厘米“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，然后重新训练决策树模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可能就会得到图 6-8 中的模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。正如我们看到的那样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，决策树有了非常大的变化“bd-box<h-char class=“bd bd-beg“>“>（原来的如图 6-2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，事实上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，由于 Scikit-Learn 的训练算法是非常随机的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，即使是相同的训练数据你也可能得到差别很大的模型“bd-box<h-char class=“bd bd-beg“>“>（除非你设置了随机数种子“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

集成学习/随机森林

假设你去随机问很多人一个很复杂的问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后把它们的答案合并起来“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。通常情况下你会发现这个合并的答案比一个专家的答案要好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这就叫做_群体智慧_“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。同样的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果你合并了一组分类器的预测“bd-box<h-char class=“bd bd-beg“>“>（像分类或者回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，你也会得到一个比单一分类器更好的预测结果“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这一组分类器就叫做集成“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>；因此“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这个技术就叫做集成学习“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一个集成学习算法就叫做集成方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

你可以训练一组决策树分类器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每一个都在一个随机的训练集上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。为了去做预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你必须得到所有单一树的预测值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后通过投票“bd-box<h-char class=“bd bd-beg“>“>（例如第六章的练习“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）来预测类别“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如一种决策树的集成就叫做随机森林“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它除了简单之外也是现今存在的最强大的机器学习算法之一“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

投票(分类)

假设你已经训练了一些分类器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每一个都有 80% 的准确率“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。你可能有了一个逻辑斯蒂回归“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、或一个 SVM“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、或一个随机森林“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，或者一个 KNN“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，或许还有更多“bd-box<h-char class=“bd bd-beg“>“>（详见图 7-1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）一个非常简单去创建一个更好的分类器的方法就是去整合每一个分类器的预测然后经过投票去预测分类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这种分类器就叫做硬投票分类器“bd-box<h-char class=“bd bd-beg“>“>（详见图 7-2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。
令人惊奇的是这种投票分类器得出的结果经常会比集成中最好的一个分类器结果更好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。事实上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，即使每一个分类器都是一个弱学习器“bd-box<h-char class=“bd bd-beg“>“>（意味着它们也就比瞎猜好点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，集成后仍然是一个强学习器“bd-box<h-char class=“bd bd-beg“>“>（高准确率“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，只要有足够数量的弱学习者“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，他们就足够多样化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.ensemble "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import RandomForestClassifier 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.ensemble "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import VotingClassifier 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.linear_model "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import LogisticRegression 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.svm "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import SVC
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> log_clf = LogisticRegression() 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> rnd_clf = RandomForestClassifier() 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> svm_clf = SVC()
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> voting_clf = VotingClassifier(estimators=[("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">'lr', log_clf), ("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">'rf', rnd_clf), 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>   ("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">'svc', svm_clf)],voting="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">'hard') 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> voting_clf.fit(X_train, y_train)

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.metrics "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import accuracy_score 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">for clf "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">in (log_clf, rnd_clf, svm_clf, voting_clf): 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>     clf.fit(X_train, y_train) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>     y_pred = clf.predict(X_test) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>     "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"built_in"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">print(clf.__class__.__name__, accuracy_score(y_test, y_pred)) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">LogisticRegression "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.864 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">RandomForestClassifier "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.872 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">SVC "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.888 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">VotingClassifier "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.896

如果所有的分类器都能够预测类别的概率“bd-box<h-char class=“bd bd-beg“>“>（例如他们有一个predict_proba()方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，那么你就可以让 sklearn 以最高的类概率来预测这个类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，平均在所有的分类器上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这种方式叫做软投票“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。他经常比硬投票表现的更好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因为它给予高自信的投票更大的权重“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。你可以通过把voting="hard"设置为voting="soft"来保证分类器可以预测类别概率“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

Bagging & Pasting

可以通过使用不同的训练算法去得到一些不同的分类器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。另一种方法就是对每一个分类器都使用相同的训练算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是在不同的训练集上去训练它们“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。有放回采样被称为装袋“bd-box<h-char class=“bd bd-beg“>“>（Bagging“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，是 bootstrap aggregating 的缩写“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。无放回采样称为粘贴“bd-box<h-char class=“bd bd-beg“>“>（pasting“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。 Bagging 和 Pasting 都允许在多个分类器上对训练集进行多次采样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但只有 Bagging 允许对同一种分类器上对训练集进行进行多次采样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。采样和训练过程如图 7-4 所示“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

当所有的分类器被训练后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，集成可以通过对所有分类器结果的简单聚合来对新的实例进行预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。聚合函数通常对分类是_统计模式_“bd-box<h-char class=“bd bd-beg“>“>（例如硬投票分类器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）或者对回归是平均“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。每一个单独的分类器在如果在原始训练集上都是高偏差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是聚合降低了偏差和方差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。通常情况下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，集成的结果是有一个相似的偏差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是对比与在原始训练集上的单一分类器来讲有更小的方差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

分类器可以通过不同的 CPU 核或其他的服务器一起被训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。相似的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，分类器也可以一起被制作“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这就是为什么 Bagging 和 Pasting 是如此流行的原因之一“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：它们的可扩展性很好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

sklearn 为 Bagging 和 Pasting 提供了一个简单的 API“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：BaggingClassifier类“bd-box<h-char class=“bd bd-beg“>“>（或者对于回归可以是BaggingRegressor“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。接下来的代码训练了一个 500 个决策树分类器的集成“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每一个都是在数据集上有放回采样 100 个训练实例下进行训练“bd-box<h-char class=“bd bd-beg“>“>（这是 Bagging 的例子“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果你想尝试 Pasting“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，就设置bootstrap=False“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。n_jobs参数告诉 sklearn 用于训练和预测所需要 CPU 核的数量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。（-1 代表着 sklearn 会使用所有空闲核“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>：

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.ensemble "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import BaggingClassifier 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.tree "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import DecisionTreeClassifier
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>bag_clf = BaggingClassifier(DecisionTreeClassifier(), n_estimators="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">500,        
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>  max_samples="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">100, bootstrap="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"literal"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">True, n_jobs=-"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>bag_clf.fit(X_train, y_train) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>y_pred = bag_clf.predict(X_test)

Bootstrap 在每个预测器被训练的子集中引入了更多的分集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以 Bagging 结束时的偏差比 Pasting 更高“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但这也意味着预测因子最终变得不相关“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，从而减少了集合的方差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。总体而言“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，Bagging 通常会导致更好的模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这就解释了为什么它通常是首选的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

对于 Bagging 来说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一些实例可能被一些分类器重复采样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但其他的有可能不会被采样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。BaggingClassifier默认采样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。BaggingClassifier默认是有放回的采样m个实例 “bd-box<h-char class=“bd bd-beg“>“>（bootstrap=True“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，其中m是训练集的大小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这意味着平均下来只有 63% 的训练实例被每个分类器采样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，剩下的 37% 个==没有被采样的训练实例就叫做 Out-of-Bag 实例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。==

在 sklearn 中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可以在训练后需要创建一个BaggingClassifier来自动评估时设置oob_score=True来自动评估“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。接下来的代码展示了这个操作“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。评估结果通过变量oob_score_来显示“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> bag_clf = BaggingClassifier(DecisionTreeClassifier(), n_estimators="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">500,bootstrap="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"literal"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">True, n_jobs=-"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1, oob_score="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"literal"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">True)
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> bag_clf.fit(X_train, y_train) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> bag_clf.oob_score_ 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.93066666666666664

随机森林

正如我们所讨论的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，随机森林是决策树的一种集成“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，通常是通过 bagging 方法“bd-box<h-char class=“bd bd-beg“>“>（有时是 pasting 方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）进行训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，通常用max_samples设置为训练集的大小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。与建立一个BaggingClassifier然后把它放入DecisionTreeClassifier相反, 你可以使用更方便的也是对决策树优化够的RandomForestClassifier“bd-box<h-char class=“bd bd-beg“>“>（对于回归是RandomForestRegressor“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。接下来的代码训练了带有 500 个树“bd-box<h-char class=“bd bd-beg“>“>（每个被限制为 16 叶子结点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）的决策森林

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.ensemble "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import RandomForestClassifier
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>rnd_clf = RandomForestClassifier(n_estimators="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">500, max_leaf_nodes="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">16, n_jobs=-"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>rnd_clf.fit(X_train, y_train)
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>y_pred_rf = rnd_clf.predict(X_test)

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.ensemble "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import RandomForestClassifier
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>rnd_clf = RandomForestClassifier(n_estimators="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">500, max_leaf_nodes="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">16, n_jobs=-"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>rnd_clf.fit(X_train, y_train)
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>y_pred_rf = rnd_clf.predict(X_test)

除了一些例外“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，RandomForestClassifier使用DecisionTreeClassifier的所有超参数“bd-box<h-char class=“bd bd-beg“>“>（决定数怎么生长“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，把BaggingClassifier的超参数加起来来控制集成本身“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
随机森林算法在树生长时引入了额外的随机“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>；与在节点分裂时需要找到最好分裂特征相反“bd-box<h-char class=“bd bd-beg“>“>（详见第六章“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，它在一个随机的特征集中找最好的特征“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它导致了树的差异性“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并且再一次用高偏差换低方差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，总的来说是一个更好的模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

极随机树
当你在随机森林上生长树时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在每个结点分裂时只考虑随机特征集上的特征“bd-box<h-char class=“bd bd-beg“>“>（正如之前讨论过的一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。相比于找到更好的特征我们可以通过使用对特征使用随机阈值使树更加随机“bd-box<h-char class=“bd bd-beg“>“>（像规则决策树一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。
这种极随机的树被简称为 Extremely Randomized Trees“bd-box<h-char class=“bd bd-beg“>“>（极随机树“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，或者更简单的称为 Extra-Tree“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。再一次用高偏差换低方差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它还使得 Extra-Tree 比规则的随机森林更快地训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因为在每个节点上找到每个特征的最佳阈值是生长树最耗时的任务之一“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

最后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果你观察一个单一决策树“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，重要的特征会出现在更靠近根部的位置“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而不重要的特征会经常出现在靠近叶子的位置“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。因此我们可以通过计算一个特征在森林的全部树中出现的平均深度来预测特征的重要性“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。sklearn 在训练后会自动计算每个特征的重要度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。你可以通过feature_importances_变量来查看结果“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.datasets "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import load_iris 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> iris = load_iris() 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> rnd_clf = RandomForestClassifier(n_estimators="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">500, n_jobs=-"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> rnd_clf.fit(iris["bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"data"], iris["bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"target"]) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">for name, score "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">in "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"built_in"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">zip(iris["bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"feature_names"], rnd_clf.feature_importances_): 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>     "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"built_in"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">print(name, score) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">sepal length (cm) "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.112492250999
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">sepal width (cm) "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.0231192882825 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">petal length (cm) "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.441030464364 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">petal width (cm) "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.423357996355

相似的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果你在 MNIST 数据及上训练随机森林分类器“bd-box<h-char class=“bd bd-beg“>“>（在第三章上介绍“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，然后画出每个像素的重要性“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可以得到图 7-6 的图片“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

Boosting

提升“bd-box<h-char class=“bd bd-beg“>“>（Boosting“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最初称为 假设增强 “bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）指的是可以将几个弱学习者组合成强学习者的集成方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。对于大多数的提升方法的思想就是按顺序去训练分类器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每一个都要尝试修正前面的分类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。现如今已经有很多的提升方法了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但最著名的就是 Adaboost“bd-box<h-char class=“bd bd-beg“>“>（适应性提升“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，是 Adaptive Boosting 的简称“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）和 Gradient Boosting“bd-box<h-char class=“bd bd-beg“>“>（梯度提升“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。让我们先从 Adaboost 说起“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

AdaBoost

使一个新的分类器去修正之前分类结果的方法就是对之前分类结果不对的训练实例多加关注“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这导致新的预测因子越来越多地聚焦于这种情况“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这是 Adaboost 使用的技术“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。举个例子“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，去构建一个 Adaboost 分类器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，第一个基分类器“bd-box<h-char class=“bd bd-beg“>“>（例如一个决策树“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）被训练然后在训练集上做预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在误分类训练实例上的权重就增加了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。第二个分类机使用更新过的权重然后再一次训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，权重更新“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，以此类推“bd-box<h-char class=“bd bd-beg“>“>（详见图 7-7“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

一旦所有的分类器都被训练后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，除了分类器根据整个训练集上的准确率被赋予的权重外“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，集成预测就非常像 Bagging 和 Pasting 了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。序列学习技术的一个重要的缺点就是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：它不能被并行化“bd-box<h-char class=“bd bd-beg“>“>（只能按步骤“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，因为每个分类器只能在之前的分类器已经被训练和评价后再进行训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。因此“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它不像 Bagging 和 Pasting 一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

让我们详细看一下 Adaboost 算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

每一个实例的权重wi初始都被设为1/m第一个分类器被训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后他的权重误差率r1在训练集上算出“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，详见公式 7-1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。其中y_tilde[j]^(i)是第j个分类器对于第i实例的预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

实例的权重

分类器的权重α[j]随后用公式 7-2 计算出来“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。其中η是超参数学习率“bd-box<h-char class=“bd bd-beg“>“>（默认为 1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。分类器准确率越高“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它的权重就越高“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果它只是瞎猜“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，那么它的权重会趋近于 0“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。然而“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果它总是出错“bd-box<h-char class=“bd bd-beg“>“>（比瞎猜的几率都低“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，它的权重会使负数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

分类器的权重

接下来实例的权重会按照公式 7-3 更新“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：误分类的实例权重会被提升“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

权重更新规则

随后所有实例的权重都被归一化“bd-box<h-char class=“bd bd-beg“>“>（例如被Σ w[i], i = 1 -> m整除“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）
最后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一个新的分类器通过更新过的权重训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，整个过程被重复“bd-box<h-char class=“bd bd-beg“>“>（新的分类器权重被计算“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，实例的权重被更新“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，随后另一个分类器被训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，以此类推“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。当规定的分类器数量达到或者最好的分类器被找到后算法就会停止“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
为了进行预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，Adaboost 通过分类器权重α[j]简单的计算了所有的分类器和权重“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。预测类别会是权重投票中主要的类别“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。（详见公式 7-4“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）其中N是分类器的数量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

AdaBoost 分类器

下来的代码训练了使用 sklearn 的AdaBoostClassifier基于 200 个决策树桩 Adaboost 分类器“bd-box<h-char class=“bd bd-beg“>“>（正如你说期待的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，对于回归也有AdaBoostRegressor“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。一个决策树桩是max_depth=1的决策树是一个单一的决策节点加上两个叶子结点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这就是AdaBoostClassifier的默认基分类器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.ensemble "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import AdaBoostClassifier
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>ada_clf = AdaBoostClassifier(DecisionTreeClassifier(max_depth="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1), n_estimators="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">200,algorithm="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"SAMME.R", learning_rate="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.5) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>>ada_clf.fit(X_train, y_train)

梯度提升

另一个非常著名的提升算法是梯度提升“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。与 Adaboost 一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，梯度提升也是通过向集成中逐步增加分类器运行的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每一个分类器都修正之前的分类结果“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。然而“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它并不像 Adaboost 那样每一次迭代都更改实例的权重“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这个方法是去使用新的分类器去拟合前面分类器预测的残差 “bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

GradientBoostingRegressor也支持指定用于训练每棵树的训练实例比例的超参数subsample“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如如果subsample=0.25“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，那么每个树都会在 25% 随机选择的训练实例上训练“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。你现在也能猜出

Stacking

本章讨论的最后一个集成方法叫做 Stacking“bd-box<h-char class=“bd bd-beg“>“>（stacked generalization 的缩写“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这个算法基于一个简单的想法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：不使用琐碎的函数“bd-box<h-char class=“bd bd-beg“>“>（如硬投票“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）来聚合集合中所有分类器的预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们为什么不训练一个模型来执行这个聚合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>？图 7-12 展示了这样一个在新的回归实例上预测的集成“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。底部三个分类器每一个都有不同的值“bd-box<h-char class=“bd bd-beg“>“>（3.1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，2.7 和 2.9“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，然后最后一个分类器“bd-box<h-char class=“bd bd-beg“>“>（叫做 blender 或者_元学习器_“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）把这三个分类器的结果当做输入然后做出最终决策“bd-box<h-char class=“bd bd-beg“>“>（3.0“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

Maxout

更快的优化器

训练一个非常大的深度神经网络可能会非常缓慢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。到目前为止“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们已经看到了四种加速训练的方法“bd-box<h-char class=“bd bd-beg“>“>（并且达到更好性能的方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>：对连接权重应用良好的初始化策略“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用良好的激活函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用批归一化以及重用预训练网络的部分 “bd-box<h-char class=“bd bd-beg“>“>（使用辅助任务或无监督学习“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。另一个速度提升的方法是使用更快的优化器 “bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而不是常规的梯度下降优化器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

动量优化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，Nesterov 加速梯度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，AdaGrad“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，RMSProp“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最后是 Adam 和 Nadam 优化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

回想一下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，梯度下降只是通过直接减去损失函数J(θ)相对于权重θ的梯度“bd-box<h-char class=“bd bd-beg“>“>（∇θJ(θ)“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，乘以学习率η来更新权重θ“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。等式是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：θ ← θ – η ∇[θ]J(θ)“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它不关心早期的梯度是什么“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果局部梯度很小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，则会非常缓慢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

动量优化很关心以前的梯度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：在每次迭代时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它将动量向量m“bd-box<h-char class=“bd bd-beg“>“>（乘以学习率η“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）与局部梯度相加“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并且通过简单地减去该动量向量来更新权重“bd-box<h-char class=“bd bd-beg“>“>（参见公式 11-4“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。换句话说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，梯度用作加速度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，不用作速度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。为了模拟某种摩擦机制“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，避免动量过大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，该算法引入了一个新的超参数β“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，简称为动量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它必须设置在 0“bd-box<h-char class=“bd bd-beg“>“>（高摩擦“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）和 1“bd-box<h-char class=“bd bd-beg“>“>（无摩擦“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）之间“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。典型的动量值是 0.9“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

可以很容易验证“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果梯度保持不变“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，则最终速度“bd-box<h-char class=“bd bd-beg“>“>（即“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，权重更新的最大大小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）等于该梯度乘以学习率η乘以1/(1-β)“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如果β = 0.9“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，则最终速度等于学习率的梯度乘以 10 倍“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因此动量优化比梯度下降快 10 倍“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>！这使动量优化比梯度下降快得多“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。特别是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我们在第四章中看到“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，当输入量具有非常不同的尺度时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，损失函数看起来像一个细长的碗“bd-box<h-char class=“bd bd-beg“>“>（见图 4-7“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。梯度下降速度很快“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但要花很长的时间才能到达底部“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。相反“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，动量优化会越来越快地滚下山谷底部“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，直到达到底部“bd-box<h-char class=“bd bd-beg“>“>（最佳“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。在不使用批归一化的深度神经网络中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，较高层往往会得到具有不同的尺度的输入“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以使用动量优化会有很大的帮助 “bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它也可以帮助滚过局部最优值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

Yurii Nesterov 在 1983 年提出的动量优化的一个小变体几乎总是比普通的动量优化更快“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。 Nesterov 动量优化或 Nesterov 加速梯度“bd-box<h-char class=“bd bd-beg“>“>（Nesterov Accelerated Gradient“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，NAG“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）的思想是测量损失函数的梯度不是在局部位置“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而是在动量方向稍微靠前“bd-box<h-char class=“bd bd-beg“>“>（见公式 11-5“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。与普通的动量优化的唯一区别在于梯度是在θ+βm而不是在θ处测量的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

再次考虑细长碗的问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：梯度下降从最陡峭的斜坡快速下降“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后缓慢地下到谷底“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果算法能够早期检测到这个问题并且纠正它的方向来指向全局最优点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，那将是非常好的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。AdaGrad 算法通过沿着最陡的维度缩小梯度向量来实现这一点“bd-box<h-char class=“bd bd-beg“>“>（见公式 11-6“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>：

第一步将梯度的平方累加到向量s中“bd-box<h-char class=“bd bd-beg“>“>（⊗符号表示元素级别相乘“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这个向量化形式相当于向量s的每个元素s[i]计算s[i] ← s[i] + (∂J(θ)/∂θ[i])^2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。换一种说法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每个s[i]累加损失函数对参数θ[i]的偏导数的平方“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果损失函数沿着第i维陡峭“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，则在每次迭代时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，s[i]将变得越来越大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

第二步几乎与梯度下降相同“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但有一个很大的不同“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：梯度向量按比例(s+ε)^0.5缩小 “bd-box<h-char class=“bd bd-beg“>“>（⊘符号表示元素分割“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，ε是避免被零除的平滑项“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，通常设置为10^(-10)“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这个向量化的形式相当于所有θ[i]同时计算

前面看到“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，AdaGrad 的风险是降速太快“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可能无法收敛到全局最优“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。RMSProp 算法通过仅累积最近迭代“bd-box<h-char class=“bd bd-beg“>“>（而不是从训练开始以来的所有梯度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）的梯度来修正这个问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它通过在第一步中使用指数衰减来实现“bd-box<h-char class=“bd bd-beg“>“>（见公式 11-7“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

如果你只看步骤 1, 2 和 5“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你会注意到 Adam 与动量优化和 RMSProp 的相似性“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。唯一的区别是第 1 步计算指数衰减的平均值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而不是指数衰减的和“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但除了一个常数因子“bd-box<h-char class=“bd bd-beg“>“>（衰减平均值只是衰减和的1 - β1倍“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）之外“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它们实际上是等效的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。动量衰减超参数β1通常初始化为 0.9“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而缩放衰减超参数β2通常初始化为 0.999“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如前所述“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，平滑项ε通常被初始化为一个很小的数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，例如10^(-7)“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

实际上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，由于 Adam 是一种自适应学习率算法“bd-box<h-char class=“bd bd-beg“>“>（如 AdaGrad 和 RMSProp“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，所以对学习率超参数η的调整较少“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。您经常可以使用默认值η= 0.001“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使 Adam 相对于梯度下降更容易使用“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

表 11-2 比较了讨论过的优化器“bd-box<h-char class=“bd bd-beg“>“>（*是差“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，**是平均“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，***是好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

正则化Regularization

有四个参数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，我可以拟合一个大象“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，五个我可以让他摆动他的象鼻“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。—— John von Neumann,cited by Enrico Fermi in Nature 427

有数千个参数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，甚至可以拟合整个动物园“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。深度神经网络通常具有数以万计的参数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，有时甚至是数百万“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。有了这么多的参数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，网络拥有难以置信的自由度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以适应各种复杂的数据集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。但是这个很大的灵活性也意味着它很容易过拟合训练集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。所以需要正则“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。第 10 章用过了最好的正则方法之一“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：早停“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。_EarlyStopping

这一节会介绍其它一些最流行的神经网络正则化技术“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：ℓ1 和 ℓ2 正则“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、丢弃和最大范数正则“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

Dropout

丢弃是深度神经网络最流行的正则化方法之一“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它由 Geoffrey Hinton 于 2012 年提出“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并在 Nitish Srivastava 等人的 2014 年论文中进一步详细描述“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并且已被证明是非常成功的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：即使是最先进的神经网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，仅仅通过增加丢弃就可以提高 1-2% 的准确度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

这是一个相当简单的算法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：在每个训练步骤中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每个神经元“bd-box<h-char class=“bd bd-beg“>“>（包括输入神经元“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但不包括输出神经元“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）都有一个暂时“丢弃”的概率p“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这意味着在这个训练步骤中它将被完全忽略“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在下一步可能会激活“bd-box<h-char class=“bd bd-beg“>“>（见图 11-9“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。超参数p称为丢弃率“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，通常设为 10% 到 50% 之间“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>；循环神经网络之间接近 20-30%“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，在卷积网络中接近 40-50%“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。训练后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，神经元不会再丢失“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

这个具有破坏性的方法竟然行得通“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这是相当令人惊讶的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果一个公司的员工每天早上被告知要掷硬币来决定是否上班“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，公司的表现会不会更好呢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>？那么“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，谁知道“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>；也许会“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>！公司显然将被迫适应这样的组织构架“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>；它不能依靠任何一个人操作咖啡机或执行任何其他关键任务“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以这个专业知识将不得不分散在几个人身上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。员工必须学会与其他的许多同事合作“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而不仅仅是其中的一小部分“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。该公司将变得更有弹性“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果一个人离开了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并没有什么区别“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。目前还不清楚这个想法是否真的可以在公司实行“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但它确实对于神经网络是可行的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。神经元被丢弃训练不能与其相邻的神经元共适应“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>；他们必须尽可能让自己变得有用“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。他们也不能过分依赖一些输入神经元;他们必须注意他们的每个输入神经元“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。他们最终对输入的微小变化会不太敏感“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。最后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你会得到一个更稳定的网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，泛化能力更强“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

CNN-CV

卷积神经网络“bd-box<h-char class=“bd bd-beg“>“>（CNN“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）起源于人们对大脑视神经的研究“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，自从 1980 年代“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，CNN 就被用于图像识别了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。最近几年“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，得益于算力提高“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、训练数据大增“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，以及第 11 章中介绍过的训练深度网络的技巧“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，CNN 在一些非常复杂的视觉任务上取得了超出人类表现的进步“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。CNN 支撑了图片搜索“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、无人驾驶汽车“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、自动视频分类“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，等等“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。另外“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，CNN 也不再限于视觉“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，比如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：语音识别和自然语言处理“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但这一章只介绍视觉应用“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

David H. Hubel 和 Torsten Wiesel 在 1958 年和 1959 年在猫的身上做了一系列研究“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，对视神经中枢做了研究“bd-box<h-char class=“bd bd-beg“>“>（并在 1981 年荣获了诺贝尔生理学或医学奖“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。特别的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，他们指出视神经中的许多神经元都有一个局部感受域“bd-box<h-char class=“bd bd-beg“>“>（local receptive field“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，也就是说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这些神经元只对有限视觉区域的刺激作反应

卷积层ConvalutionalLayer

卷积层是 CNN 最重要的组成部分“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：第一个卷积层的神经元“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，不是与图片中的每个像素点都连接“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而是只连着局部感受野的像素“bd-box<h-char class=“bd bd-beg“>“>（见图 14-2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。同理“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，第二个卷积层中的每个神经元也只是连着第一层中一个小方形内的神经元“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这种架构可以让第一个隐藏层聚焦于小的低级特征“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后在下一层组成大而高级的特征

神经元的权重可以表示为感受野大小的图片“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，图 14-5 展示了两套可能的权重“bd-box<h-char class=“bd bd-beg“>“>（称为权重“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，或卷积核“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。第一个是黑色的方形“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，中央有垂直白线“bd-box<h-char class=“bd bd-beg“>“>（7 × 7的矩阵“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，除了中间的竖线都是 1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，其它地方是 0“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>；使用这个矩阵“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，神经元只能注意到中间的垂直线“bd-box<h-char class=“bd bd-beg“>“>（因为其它地方都乘以 0 了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。第二个过滤器也是黑色的方形“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但是中间是水平的白线“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。使用这个权重的神经元只会注意中间的白色水平线“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

如果卷积层的所有神经元使用同样的垂直过滤器“bd-box<h-char class=“bd bd-beg“>“>（和同样的偏置项“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，给神经网络输入图 14-5 中最底下的图片“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，卷积层输出的是左上的图片“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。可以看到“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，图中垂直的白线得到了加强“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，其余部分变模糊了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。相似的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，右上的图是所有神经元都是用水平线过滤器的结果“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，水平的白线加强了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，其余模糊了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。因此“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一层的全部神经元都用一个过滤器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，就能输出一个特征映射“bd-box<h-char class=“bd bd-beg“>“>（feature map“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，特征映射可以高亮图片中最为激活过滤器的区域“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。当然“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，不用手动定义过滤器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：卷积层在训练中可以自动学习对任务最有用的过滤器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，上面的层则可以将简单图案组合为复杂图案“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

简单起见“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，前面都是将每个卷积层的输出用 2D 层来表示的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但真实的卷积层可能有多个过滤器“bd-box<h-char class=“bd bd-beg“>“>（过滤器数量由你确定“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，每个过滤器会输出一个特征映射“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以表示成 3D 更准确

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">下面代码使用 Scikit-Learn 的`load_sample_image()`加载了两张图片"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">，一张是中国的寺庙"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">，另一张是花"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">，创建了两个过滤器"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">，应用到了两张图片上"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">，最后展示了一张特征映射"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">：
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">from sklearn.datasets "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"keyword"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">import load_sample_image
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"comment"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"># 加载样本图片
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">china = load_sample_image("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"china.jpg") / "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">255
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">flower = load_sample_image("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"flower.jpg") / "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">255
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">/images = np.array([china, flower])
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">batch_size, height, width, channels = /images.shape
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"comment"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"># 创建两个过滤器
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">filters = np.zeros(shape=("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">7, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">7, channels, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">2), dtype=np.float32)
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">filters[:, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">3, :, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0] = "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1  "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"comment"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"># 垂直线
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">filters["bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">3, :, :, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1] = "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1  "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"comment"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"># 水平线
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">outputs = tf.nn.conv2d(/images, filters, strides="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1, padding="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"same")
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">plt.imshow(outputs["bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0, :, :, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1], cmap="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"gray") "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"comment"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"># 画出第 1 张图的第 2 个特征映射
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">plt.show() 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">这个例子中"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">，我们手动定义了过滤器"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">，但在真正的 CNN 中"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">，一般将过滤器定义为可以训练的变量"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">，好让神经网络学习哪个过滤器的效果最好"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">。使用`keras.layers.Conv2D`层"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">：
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">conv = keras.layers.Conv2D(filters="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">32, kernel_size="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">3, strides="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1,
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">                           padding="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"same", activation="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"relu") 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">

tf.nn.conv2d()函数这一行“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，再多说说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：

/images是一个输入的小批次“bd-box<h-char class=“bd bd-beg“>“>（4D 张量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。
filters是过滤器的集合“bd-box<h-char class=“bd bd-beg“>“>（也是 4D 张量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。
strides等于 1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，也可以是包含 4 个元素的 1D 数组“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，中间的两个元素是垂直和水平步长“bd-box<h-char class=“bd bd-beg“>“>（s[h]和s[w]“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，第一个和最后一个元素现在必须是 1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。以后可以用来指定批次步长“bd-box<h-char class=“bd bd-beg“>“>（跳过实例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）和通道步长“bd-box<h-char class=“bd bd-beg“>“>（跳过前一层的特征映射或通道“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。
padding必须是"same"或"valid"“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：
如果设为"same"“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，卷积层会使用零填充“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。输出的大小是输入神经元的数量除以步长“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，再取整“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：如果输入大小是 13“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，步长是 5“bd-box<h-char class=“bd bd-beg“>“>（见图 14-7“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，则输出大小是 3“bd-box<h-char class=“bd bd-beg“>“>（13 / 5 = 2.6“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，再向上圆整为 3“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，零填充尽量在输入上平均添加“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。当strides=1时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，层的输出会和输入有相同的空间维度“bd-box<h-char class=“bd bd-beg“>“>（宽和高“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，这就是same的来历“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
如果设为"valid"“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，卷积层就不使用零填充“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，取决于步长“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可能会忽略图片的输入图片的底部或右侧的行和列“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，见图 14-7“bd-box<h-char class=“bd bd-beg“>“>（简单举例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，只是显示了水平维度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这意味着每个神经元的感受野位于严格确定的图片中的位置“bd-box<h-char class=“bd bd-beg“>“>（不会越界“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，这就是valid的来历“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

CNN 的另一个问题是卷积层需要很高的内存“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。特别是在训练时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因为反向传播需要所有前向传播的中间值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

池化层

明白卷积层的原理了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，池化层就容易多了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。池化层的目的是对输入图片做降采样“bd-box<h-char class=“bd bd-beg“>“>（即“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，收缩“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，以降低计算负载“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、内存消耗和参数的数量“bd-box<h-char class=“bd bd-beg“>“>（降低过拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

和卷积层一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，池化层中的每个神经元也是之和前一层的感受野里的有限个神经元相连“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。和前面一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，必须定义感受野的大小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、步长和填充类型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。但是“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，池化神经元没有权重“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它所要做的是使用聚合函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，比如最大或平均“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，对输入做聚合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。图 14-8 展示了最为常用的最大池化层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在这个例子中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用了一个2 × 2的池化核“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，步长为 2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，没有填充“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。只有感受野中的最大值才能进入下一层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，其它的就丢弃了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

除了可以减少计算“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、内存消耗“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、参数数量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最大池化层还可以带来对小偏移的不变性

在 CNN 中每隔几层就插入一个最大池化层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以带来更大程度的平移不变性“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。另外“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最大池化层还能带来一定程度的旋转不变性和缩放不变性“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。当预测不需要考虑平移“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、旋转和缩放时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，比如分类任务“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，不变性可以有一定益处“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

要创建平均池化层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，则使用AvgPool2D“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。平均池化层和最大池化层很相似“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但计算的是感受野的平均值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。平均池化层在过去很流行“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但最近人们使用最大池化层更多“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因为最大池化层的效果更好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

池化层还可以沿着深度方向做计算“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这可以让 CNN 学习到不同特征的不变性“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。比如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。CNN 可以学习多个过滤器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每个过滤器检测一个相同的图案的不同旋转“bd-box<h-char class=“bd bd-beg“>“>（比如手写字“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，见图 14-10“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，深度池化层可以使输出相同“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。CNN 还能学习其它的不变性“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：厚度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、明亮度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、扭曲“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、颜色“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，等等“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

数据增强

数据增强是通过生成许多训练实例的真实变种“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，来人为增大训练集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。因为可以降低过拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，成为了一种正则化方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。生成出来的实例越真实越好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：最理想的情况“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，人们无法区分增强图片是原生的还是增强过的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。简单的添加白噪声没有用“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，增强修改要是可以学习的“bd-box<h-char class=“bd bd-beg“>“>（白噪声不可学习“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，==可以轻微偏移“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、旋转“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、缩放原生图“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，再添加到训练集中==“bd-box<h-char class=“bd bd-beg“>“>（见图 14-12“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这么做可以使模型对位置“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、方向和物体在图中的大小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，有更高的容忍度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果想让模型对不同光度有容忍度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以生成对比度不同的照片“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。通常“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，==还可以水平翻转图片“bd-box<h-char class=“bd bd-beg“>“>（文字不成“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、不对称物体也不成“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）==“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。通过这些变换“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以极大的增大训练集“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

CNN的典型架构

CNN 的典型架构是将几个卷积层叠起来“bd-box<h-char class=“bd bd-beg“>“>（每个卷积层后面跟着一个 ReLU 层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，然后再叠一个池化层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后再叠几个卷积层“bd-box<h-char class=“bd bd-beg“>“>（+ReLU“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，接着再一个池化层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，以此类推“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。图片在流经神经网络的过程中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，变得越来越小“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但得益于卷积层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，却变得越来越深“bd-box<h-char class=“bd bd-beg“>“>（特征映射变多了“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，见图 14-11“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在 CNN 的顶部“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，还有一个常规的前馈神经网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，由几个全连接层“bd-box<h-char class=“bd bd-beg“>“>（+ReLU“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）组成“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最终层输出预测“bd-box<h-char class=“bd bd-beg“>“>（比如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，一个输出类型概率的 softmax 层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">model = keras.models.Sequential([
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Conv2D("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">64, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">7, activation="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"relu", padding="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"same",
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">                        input_shape=["bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">28, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">28, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1]),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.MaxPooling2D("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">2),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Conv2D("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">128, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">3, activation="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"relu", padding="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"same"),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Conv2D("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">128, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">3, activation="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"relu", padding="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"same"),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.MaxPooling2D("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">2),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Conv2D("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">256, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">3, activation="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"relu", padding="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"same"),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Conv2D("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">256, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">3, activation="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"relu", padding="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"same"),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.MaxPooling2D("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">2),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Flatten(),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Dense("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">128, activation="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"relu"),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Dropout("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.5),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Dense("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">64, activation="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"relu"),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Dropout("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.5),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Dense("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">10, activation="bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"string"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"softmax")
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">])

我们先看看经典的 LeNet-5 架构“bd-box<h-char class=“bd bd-beg“>“>（1998“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，然后看看三个 ILSVRC 竞赛的冠军“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：AlexNet“bd-box<h-char class=“bd bd-beg“>“>（2012“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>、GoogLeNet“bd-box<h-char class=“bd bd-beg“>“>（2014“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>、ResNet“bd-box<h-char class=“bd bd-beg“>“>（2015“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

LeNet-5 也许是最广为人知的 CNN 架构“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。前面提到过“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它是由 Yann LeCun 在 1998 年创造出来的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，被广泛用于手写字识别“bd-box<h-char class=“bd bd-beg“>“>（MNIST“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。它的结构如下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：

AlexNet 和 LeNet-5 很相似“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，只是更大更深“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，是首个将卷积层堆叠起来的网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而不是在每个卷积层上再加一个池化层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。为了降低过拟合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，作者使用了两种正则方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。首先“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，F8 和 F9 层使用了丢弃“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，丢弃率为 50%“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。其次“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，他们通过随机距离偏移训练图片“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、水平翻转“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、改变亮度“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，做了数据增强“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

GoogLeNet 架构能取得这么大的进步“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，很大的原因是它的网络比之前的 CNN 更深“bd-box<h-char class=“bd bd-beg“>“>（见图 14-14“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这归功于被称为创始模块“bd-box<h-char class=“bd bd-beg“>“>（inception module“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）的子网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它可以让 GoogLeNet 可以用更高的效率使用参数

ResNet 的使用了极深的卷积网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，共 152 层“bd-box<h-char class=“bd bd-beg“>“>（其它的变体有 1450 或 152 层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。反映了一个总体趋势“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：模型变得越来越深“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，参数越来越少“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。训练这样的深度网络的方法是使用跳连接“bd-box<h-char class=“bd bd-beg“>“>（也被称为快捷连接“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>：输入信号添加到更高层的输出上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

训练神经网络时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，目标是使网络可以对目标函数h(x)建模“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果将输入x添加给网络的输出“bd-box<h-char class=“bd bd-beg“>“>（即“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，添加一个跳连接“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，则网络就要对f(x) = h(x) – x建模“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而不是h(x)“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这被称为残差学习“bd-box<h-char class=“bd bd-beg“>“>（见图 14-15“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

目标检测

分类并定位图片中的多个物体的任务被称为目标检测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。几年之前“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使用的方法还是用定位单一目标的 CNN“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后将其在图片上滑动

用这个简单的方法来做目标检测的效果相当不错“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但需要运行 CNN 好几次“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以很慢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。幸好“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，有一个更快的方法来滑动 CNN“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：使用全卷积网络“bd-box<h-char class=“bd bd-beg“>“>（fully convolutional network“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，FCN“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

YOLO 是一个非常快且准确的目标检测框架“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，是 Joseph Redmon 在 2015 年的一篇论文中提出的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，2016 年优化为 YOLOv2“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，2018 年优化为 YOLOv3“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。速度快到甚至可以在实时视频中运行

语义分割

在语义分割中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每个像素根据其所属的目标来进行分类“bd-box<h-char class=“bd bd-beg“>“>（例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，路“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、汽车“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、行人“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>、建筑物“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，等等“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，见图 14-26“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。注意“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，相同类的不同目标是不做区分的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，分割图片的右侧的所有自行车被归类为一坨像素“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这个任务的难点是当图片经过常规 CNN 时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，会逐渐丢失空间分辨率“bd-box<h-char class=“bd bd-beg“>“>（因为有的层的步长大于 1“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>；因此“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，常规的 CNN 可以检测出图片的左下有一个人“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但不知道准确的位置“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

RNN

RNN 不是唯一能处理序列数据的神经网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：对于小序列“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，常规紧密网络也可以“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>；对于长序列“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，比如音频或文本“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，卷积神经网络也可以“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。我们会讨论这两种方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，本章最后会实现一个 WaveNet“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：这是一种 CNN 架构“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以处理上万个时间步的序列“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在第 16 章“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，还会继续学习 RNN“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如何使用 RNN 来做自然语言处理“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，和基于注意力机制的新架构“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

我们主要关注的是前馈神经网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，激活仅从输入层到输出层的一个方向流动“bd-box<h-char class=“bd bd-beg“>“>（附录 E 中的几个网络除外“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。循环神经网络看起来非常像一个前馈神经网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，除了它也有连接指向后方“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。让我们看一下最简单的 RNN“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，由一个神经元接收输入“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，产生一个输出“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，并将输出发送回自己“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，如图 15-1“bd-box<h-char class=“bd bd-beg“>“>（左“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）所示“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

每个循环神经元有两组权重“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：一组用于输入x[t]“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，另一组用于前一时间步长y[t - 1]的输出“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。我们称这些权重向量为w[x]和w[y]“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。如果考虑的是整个循环神经元层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以将所有权重向量放到两个权重矩阵中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，W[x]和W[y]“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。整个循环神经元层的输出可以用公式 15-1 表示“bd-box<h-char class=“bd bd-beg“>“>（b是偏差项“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，φ(·)是激活函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，例如 ReLU“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

一般情况下“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，时间步t的单元状态“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，记为h[t]“bd-box<h-char class=“bd bd-beg“>“>（h代表“隐藏”“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，是该时间步的某些输入和前一时间步状态的函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：h[t] = f(h[t - 1], x[t])“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。其在时间步t的输出“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，表示为y[t]“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，也和前一状态和当前输入的函数有关“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

RNN 可以同时输入序列并输出序列“bd-box<h-char class=“bd bd-beg“>“>（见图 15-4“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，左上角的网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这种序列到序列的网络可以有效预测时间序列“bd-box<h-char class=“bd bd-beg“>“>（如股票价格“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>：输入过去N天价格“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，则输出向未来移动一天的价格“bd-box<h-char class=“bd bd-beg“>“>（即“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，从N - 1天前到明天“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

或者“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可以向网络输入一个序列“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，忽略除最后一项之外的所有输出“bd-box<h-char class=“bd bd-beg“>“>（图 15-4 右上角的网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。换句话说“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这是一个序列到向量的网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，你可以向网络输入与电影评论相对应的单词序列“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，网络输出情感评分“bd-box<h-char class=“bd bd-beg“>“>（例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，从-1 [讨厌]到+1 [喜欢]“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

相反“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以向网络一遍又一遍输入相同的向量“bd-box<h-char class=“bd bd-beg“>“>（见图 15-4 的左下角“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，输出一个序列“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这是一个向量到序列的网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，输入可以是图像“bd-box<h-char class=“bd bd-beg“>“>（或是 CNN 的结果“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，输出是该图像的标题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

给网络输入一种语言的一句话“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，编码器会把这个句子转换成单一的向量表征“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，然后解码器将这个向量解码成另一种语言的句子“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这种称为编码器 - 解码器的两步模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，比用单个序列到序列的 RNN 实时地进行翻译要好得多“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，因为句子的最后一个单词可以影响翻译的第一句话“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，所以你需要等到听完整个句子才能翻译“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。第 16 章还会介绍如何实现编码器-解码器“bd-box<h-char class=“bd bd-beg“>“>（会比图 15-4 中复杂“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

训练 RNN 诀窍是在时间上展开“bd-box<h-char class=“bd bd-beg“>“>（就像我们刚刚做的那样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，然后只要使用常规反向传播“bd-box<h-char class=“bd bd-beg“>“>（见图 15-5“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。这个策略被称为时间上的反向传播“bd-box<h-char class=“bd bd-beg“>“>（BPTT“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

假设你在研究网站每小时的活跃用户数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，或是所在城市的每日气温“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，或公司的财务状况“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，用多种指标做季度衡量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在这些任务中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，数据都是一个序列“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每步有一个或多个值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这被称为时间序列“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

使用 RNN 之前“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最好有基线指标“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，否则做出来的模型可能比基线模型还糟“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。例如“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，最简单的方法“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，是预测每个序列的最后一个值“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这个方法被称为朴素预测“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，有时很难被超越“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。在这个例子中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它的均方误差为 0.020“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：

另一个简单的方法是使用全连接网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。因为结果要是打平的特征列表“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，需要加一个Flatten层“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。使用简单线性回归模型“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，使预测值是时间序列中每个值的线性组合“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：

"bd bd-beg

bd-box

bd bd-beg

highlight python"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg

bd-box

bd bd-beg

code"bd bd-beg

bd-box

bd bd-beg

"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> y_pred = X_valid[:, -"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1]
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"meta"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">>>> np.mean(keras.losses.mean_squared_error(y_valid, y_pred))
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">0.020211367 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">model = keras.models.Sequential([
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Flatten(input_shape=["bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">50, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1]),
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">    keras.layers.Dense("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1)
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">]) 
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">model = keras.models.Sequential([
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">  keras.layers.SimpleRNN("bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1, input_shape=["bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"literal"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">None, "bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"number"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">1])
"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">"line"bd bd-beg">"bd-box"bd bd-beg">">"bd bd-beg">">])

将多个神经元的层堆起来“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，见图 15-7“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。就形成了深度 RNN“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

在训练长序列的 RNN 模型时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，必须运行许多时间步“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，展开的 RNN 变成了一个很深的网络“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。正如任何深度神经网络一样“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它面临不稳定梯度问题“bd-box<h-char class=“bd bd-beg“>“>（第 11 章讨论过“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，使训练无法停止“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，或训练不稳定“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。另外“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，当 RNN 处理长序列时“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，RNN 会逐渐忘掉序列的第一个输入“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。下面就来看看这两个问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，先是第一个问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

很多之前讨论过的缓解不稳定梯度的技巧都可以应用在 RNN 中“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：好的参数初始化方式“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，更快的优化器“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，丢弃“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，等等“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。但是非饱和激活函数“bd-box<h-char class=“bd bd-beg“>“>（如 ReLU“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）的帮助不大“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>；事实上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，它会导致 RNN 更加不稳定“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。为什么呢“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>？假设梯度下降更新了权重“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以令第一个时间步的输出提高“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。因为每个时间步使用的权重相同“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，第二个时间步的输出也会提高“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，这样就会导致输出爆炸 —— 不饱和激活函数不能阻止这个问题“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。要降低爆炸风险“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，可以使用更小的学习率“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，更简单的方法是使用一个饱和激活函数“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，比如双曲正切函数“bd-box<h-char class=“bd bd-beg“>“>（这就解释了为什么 tanh 是默认选项“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。

另外“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，批归一化也没什么帮助“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。事实上“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，不能在时间步骤之间使用批归一化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，只能在循环层之间使用“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。更加准确点“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，技术上可以将 BN 层添加到记忆单元上“bd-box<h-char class=“bd bd-beg“>“>（后面会看到“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，这样就可以应用在每个时间步上了“bd-box<h-char class=“bd bd-beg“>“>（既对输入使用“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，也对前一步的隐藏态使用“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>。
使用tf.keras在一个简单记忆单元中实现层归一化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。
另一种归一化的形式效果好些“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：层归一化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。它是由 Jimmy Lei Ba 等人在 2016 年的一篇论文中提出的“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>：它跟批归一化很像“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，但不是在批次维度上做归一化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，而是在特征维度上归一化“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这么做的一个优势是可以独立对每个实例“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，实时计算所需的统计量“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。这还意味着训练和测试中的行为是一致的“bd-box<h-char class=“bd bd-beg“>“>（这点和 BN 相反“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）<h-char class=“bd bd-beg“>，且不需要使用指数移动平均来估计训练集中所有实例的特征统计“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

由于数据在 RNN 中流动时会经历转换“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，每个时间步都损失了一定信息“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。一定时间后“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>，第一个输入实际上会在 RNN 的状态中消失“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>。

Keyspire's Peakory

目录

人工智能笔记

机器学习概览

监督学习和非监督学习

强化学习

批量和在线学习

批量学习

在线学习

线性回归

正规方程求解MSE

如何训练模型?梯度下降

批量梯度下降

随机梯度下降

小批量梯度下降

学习曲线/过拟合/欠拟合

误差来源分析

正则化

岭回归

Lasso回归

弹性网络<span class=<h-char class=“bd bd-beg“>“bd-box<h-char class=“bd bd-beg“>“>（ElasticNet<span class=<h-char class=“bd bd-beg“>“bd-box<h-char class=“bd bd-beg“>“><h-char class=“bd bd-beg“>）

_EarlyStopping

机器学习基础

支持向量机SVM

线性SVM分类

软间隔分类

非线性SVM分类

多项式核

增加相似特征

高斯 RBF 核

计算复杂度

SVM回归

决策树

分类

_CART训练算法

正则化

回归

不稳定性

集成学习/随机森林

投票(分类)

Bagging & Pasting

随机森林

Boosting

AdaBoost

梯度提升

Stacking

Maxout

更快的优化器

正则化Regularization

Dropout

CNN-CV

卷积层ConvalutionalLayer

池化层

数据增强

CNN的典型架构

目标检测

语义分割

RNN