本文结合50层深度残差网络的实现学习何博士的大作-Deep Residual Learning for Image Recognition。理论上&＃xff0c;深层网络结构包含了浅层网络结构所有可能的解空间&＃xff0c;但是实际网络训练中&＃xff0c;随着网络深度的增加&＃xff0c;网络的准确度出现饱和&＃xff0c;甚至下降的现象&＃xff0c;这个现象可以在下图直观看出来&＃xff1a;56层的网络比20层网络效果还要差。但是这种退化并不是因为过拟合导致的&＃xff0c;因为56层的神经网络的训练误差同样高。
这就是神经网络的退化现象。何博士提出的残差学习的方法解决了解决了神经网络的退化问题&＃xff0c;在深度学习领域取得了巨大的成功。
1.Residual Networks各个深度的神经网络的结构如下:
50层网络的结构实际上是把34层网络的2个3x3的卷积层替换成3个卷积层&＃xff1a;1x1、3x3、1x1,可以看到50层的网络相对于34层的网络&＃xff0c;效果上有不小的提升。
代码实现
ResNet 50代码实现的网络结构与上图50层的网络架构完全一致。对于深度较深的神经网络&＃xff0c;BN必不可少&＃xff0c;关于BN的介绍和实现可以参考以前的文章。
class ResNet50(object):
def __init__(self, inputs, num_classes&＃61;1000, is_training&＃61;True,
scope&＃61;"resnet50"):
self.inputs &＃61;inputs
self.is_training &＃61; is_training
self.num_classes &＃61; num_classes
with tf.variable_scope(scope):
# construct the model
net &＃61; conv2d(inputs, 64, 7, 2, scope&＃61;"conv1") # -> [batch, 112, 112, 64]
net &＃61; tf.nn.relu(batch_norm(net, is_training&＃61;self.is_training, scope&＃61;"bn1"))
net &＃61; max_pool(net, 3, 2, scope&＃61;"maxpool1") # -> [batch, 56, 56, 64]
net &＃61; self._block(net, 256, 3, init_stride&＃61;1, is_training&＃61;self.is_training,
scope&＃61;"block2") # -> [batch, 56, 56, 256]
net &＃61; self._block(net, 512, 4, is_training&＃61;self.is_training, scope&＃61;"block3")
# -> [batch, 28, 28, 512]
net &＃61; self._block(net, 1024, 6, is_training&＃61;self.is_training, scope&＃61;"block4")
# -> [batch, 14, 14, 1024]
net &＃61; self._block(net, 2048, 3, is_training&＃61;self.is_training, scope&＃61;"block5")
# -> [batch, 7, 7, 2048]
net &＃61; avg_pool(net, 7, scope&＃61;"avgpool5") # -> [batch, 1, 1, 2048]
net &＃61; tf.squeeze(net, [1, 2], name&＃61;"SpatialSqueeze") # -> [batch, 2048]
self.logits &＃61; fc(net, self.num_classes, "fc6") # -> [batch, num_classes]
self.predictions &＃61; tf.nn.softmax(self.logits)
2.Building Block每个Block中往往包含多个子Block&＃xff0c;每个子Block又有多个卷积层组成。每个Block的第一个子Block的第一个卷积层的stride&＃61;2&＃xff0c;完成Feature Map的下采样的工作。
代码实现
def _block(self, x, n_out, n, init_stride&＃61;2, is_training&＃61;True, scope&＃61;"block"):
with tf.variable_scope(scope):
h_out &＃61; n_out // 4
out &＃61; self._bottleneck(x, h_out, n_out, stride&＃61;init_stride,
is_training&＃61;is_training, scope&＃61;"bottlencek1")
for i in range(1, n):
out &＃61; self._bottleneck(out, h_out, n_out, is_training&＃61;is_training,
scope&＃61;("bottlencek%s" % (i &＃43; 1)))
return out
3. Bottleneck Architectures在更深层(esNet-50/101/152)的神经网络中为了节省计算耗时&＃xff0c; 作者对神经网络的架构进行了改造&＃xff0c;将原有的两层3x3卷积层改造为三层卷积层&＃xff1a;1x1&＃xff0c;3x3&＃xff0c;1x1。
The three layers are 1×1, 3×3, and 1×1 convolutions, where the 1×1 layers are responsible for reducing and then increasing (restoring)dimensions, leaving the 3×3 layer a bottleneck with smaller input/output dimensions。
代码实现:
x: 是输入数据&＃xff0c;格式为[BatchSize, ImageHeight&＃xff0c;ImageWidth, ChannelNum]&＃xff1b;
h_out: 卷积核个数&＃xff1b;
n_out: Block的输出的卷积核个数&＃xff1b;
stride: 卷积步长&＃xff1b;
is_training: 用于Batch Normalization&＃xff1b;
def _bottleneck(self, x, h_out, n_out, stride&＃61;None, is_training&＃61;True, scope&＃61;"bottleneck"):
""" A residual bottleneck unit"""
n_in &＃61; x.get_shape()[-1]
if stride is None:
stride &＃61; 1 if n_in &＃61;&＃61; n_out else 2
with tf.variable_scope(scope):
h &＃61; conv2d(x, h_out, 1, stride&＃61;stride, scope&＃61;"conv_1")
h &＃61; batch_norm(h, is_training&＃61;is_training, scope&＃61;"bn_1")
h &＃61; tf.nn.relu(h)
h &＃61; conv2d(h, h_out, 3, stride&＃61;1, scope&＃61;"conv_2")
h &＃61; batch_norm(h, is_training&＃61;is_training, scope&＃61;"bn_2")
h &＃61; tf.nn.relu(h)
h &＃61; conv2d(h, n_out, 1, stride&＃61;1, scope&＃61;"conv_3")
h &＃61; batch_norm(h, is_training&＃61;is_training, scope&＃61;"bn_3")
if n_in !&＃61; n_out:
shortcut &＃61; conv2d(x, n_out, 1, stride&＃61;stride, scope&＃61;"conv_4")
shortcut &＃61; batch_norm(shortcut, is_training&＃61;is_training, scope&＃61;"bn_4")
else:
shortcut &＃61; x
return tf.nn.relu(shortcut &＃43; h)
4. ShortcutsIdentity Mapping是深度残差网络的一个核心思想&＃xff0c;深度残差网络中Building Block表达公式如下:
x是Layer Input&＃xff0c; y是未经过Relu激活函数的Layer Output&＃xff0c; 是待学习的残差映射。
上式仅仅能处理F(x, wi)和x维度相同的情况&＃xff0c;当二者维度不同的情况下应该怎么处理呢&＃xff1f;
作者提出了两种处理方式: zero padding shortcut和 projection shortcut。并在实验中构造三种shortcut的方式&＃xff1a;
A) 当数据维度增加时&＃xff0c;采用zero padding进行数据填充&＃xff1b;
B) 当数据维度增加时&＃xff0c;采用projection的方式&＃xff1b;数据维度不变化时&＃xff0c;直接使用恒等映射&＃xff1b;
C) 数据维度增加与否都采用projection的方式&＃xff1b;
三种方式的对比效果如下:
可以看到效果排序如下&＃xff1a; A
5.其它辅助函数的实现
5.1 变量初始化
fc_initializer &＃61; tf.contrib.layers.xavier_initializer
conv2d_initializer &＃61; tf.contrib.layers.xavier_initializer_conv2d
5.2 创建变量的辅助函数
# create weight variable
def create_var(name, shape, initializer, trainable&＃61;True):
return tf.get_variable(name, shape&＃61;shape, dtype&＃61;tf.float32,
initializer&＃61;initializer, trainable&＃61;trainable)
5.3 卷积辅助函数
# conv2d layer
def conv2d(x, num_outputs, kernel_size, stride&＃61;1, scope&＃61;"conv2d"):
num_inputs &＃61; x.get_shape()[-1]
with tf.variable_scope(scope):
kernel &＃61; create_var("kernel
京公网安备 11010802041100号