pytorch从入门到精通

GitHub - mint-lab/dl_tutorial: Deep Learning Tutorial with PyTorch:slides

先做减法&＃xff0c;具体例子带你了解torch使用的基本套路&＃xff08;分类和时间序列小例子&＃xff09;

pytorch中Tensor和Variable有什么区别&＃xff1f;PyTorch内部机制

DataSet要实现哪几个函数&＃xff1f;

二元分类为什么不能用MSE做为损失函数?

怎么获取网络的计算量和模型大小&＃xff1f;有哪几种模型加载和保存方式&＃xff1f;pytorch-summary, flops-counter.pytorch

有哪些提高pytorch 训练速度的trick? 预处理加速albumentations

PyTorch 深度学习:60分钟快速入门 ImageNet training in PyTorch, 训练一个图像分类模型

PyTorch源码解读之torch.utils.data.DataLoader、torchvision.transforms、torchvision.models,

PyTorch实战指南、trick 集锦、半小时学会 PyTorch Hook、深度学习模型转换与部署那些事(含ONNX格式详细分析)

详解Pytorch中的网络构造、resnet50pytorch, Autograd

PyTorch分布式训练简明教程: horovod

部署PyTorch模型到终端: demonet

以训练最简单的mnist为例&＃xff0c;完整的例子为:

import os
import numpy as np
import cv2import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvisionfrom tqdm import tqdm
from torchsummary import summarydevice &＃61; torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")transform &＃61; torchvision.transforms.Compose([torchvision.transforms.ToTensor(),torchvision.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),])trainset &＃61; torchvision.datasets.MNIST(root&＃61;&＃39;./data/MNIST&＃39;, train&＃61;True,download&＃61;True, transform&＃61;transform)
trainloader &＃61; torch.utils.data.DataLoader(trainset, batch_size&＃61;256,shuffle&＃61;True, num_workers&＃61;8)testset &＃61; torchvision.datasets.MNIST(root&＃61;&＃39;./data/MNIST&＃39;, train&＃61;False,download&＃61;True, transform&＃61;transform)
testloader &＃61; torch.utils.data.DataLoader(testset, batch_size&＃61;8,shuffle&＃61;False, num_workers&＃61;8)def readtest():for images, _ in tqdm(trainloader):img &＃61; torchvision.utils.make_grid(images,4)img &＃61; img.numpy()*2&＃43;0.5img &＃61; np.transpose(img, (1, 2, 0))cv2.imshow("img",img)cv2.waitKey()class Net(nn.Module):def __init__(self):super(Net, self).__init__()self.conv1 &＃61; nn.Conv2d(1, 6, 5)self.conv2 &＃61; nn.Conv2d(6, 16, 5)self.fc1 &＃61; nn.Linear(16 * 4 * 4, 120)self.fc2 &＃61; nn.Linear(120, 84)self.fc3 &＃61; nn.Linear(84, 10)def forward(self, x):x &＃61; F.max_pool2d(F.relu(self.conv1(x)), 2)x &＃61; F.max_pool2d(F.relu(self.conv2(x)), 2)x &＃61; x.view(-1, self.num_flat_features(x))x &＃61; F.relu(self.fc1(x))x &＃61; F.relu(self.fc2(x))x &＃61; self.fc3(x)return xdef num_flat_features(self, x):size &＃61; x.size()[1:]num_features &＃61; 1for s in size:num_features *&＃61; sreturn num_featuresdef val(net):net.eval()correct &＃61; 0total &＃61; 0pbar &＃61; tqdm(testloader)for images, labels in pbar:images &＃61; images.to(device)labels &＃61; labels.to(device)outputs &＃61; net(images)_, predicted &＃61; torch.max(outputs.data, 1)total &＃43;&＃61; labels.size(0)correct &＃43;&＃61; (predicted &＃61;&＃61; labels).sum()acc &＃61; correct * 100.0 / totalpbar.set_description("acc: {acc:.2f}".format(acc&＃61;acc))acc &＃61; correct * 100.0 / totalprint("val acc&＃61;{acc:.3f}".format(acc&＃61;acc))return accdef train(net):bestacc &＃61; 0if os.path.exists("best.pth"):model &＃61; torch.load("best.pth")net.load_state_dict(model)bestacc &＃61; val(net)print("Resuming from acc &＃61; {acc:.3f}".format(acc &＃61; bestacc))optimizer &＃61; torch.optim.SGD(net.parameters(), lr&＃61;0.001, momentum&＃61;0.9)criterion &＃61; nn.CrossEntropyLoss()epochs &＃61; 100000for epoch in range(epochs):print("Epoch: "&＃43;str(epoch))net.train()pbar &＃61; tqdm(trainloader)for images, labels in pbar:images &＃61; images.to(device)labels &＃61; labels.to(device)optimizer.zero_grad()outputs &＃61; net(images)_, predicted &＃61; torch.max(outputs.data, 1)acc &＃61; (predicted &＃61;&＃61; labels).sum() * 100.0 / labels.size(0)loss &＃61; criterion(outputs,labels)loss.backward()optimizer.step()pbar.set_description("loss: {loss:.3f},acc: {acc:.2f}".format(loss&＃61;loss,acc&＃61;acc))acc &＃61; val(net)if acc > bestacc:torch.save(net.state_dict(),"best.pth")bestacc &＃61; accprint("best improve to {acc:.3f}".format(acc&＃61;acc))torch.save(net.state_dict(),"last.pth")def main():net &＃61; Net()net.to(device)#summary(net,(1,28,28))#readtest()train(net)if __name__&＃61;&＃61;"__main__":main()

pytorch2caffe ShuffleNet_V2_pytorch_caffe

首推PytorchToCaffe, 其支持 0.3-1.*版本的转换&＃xff0c;注意1.1有些BUG暂不支持&＃xff1b;还有就是不支持双线性插值的上采样层&＃xff0c;这个在分割模型里用的比较多&＃xff1b;还有就是torchvison的版本一定得是0.2&＃xff0c;不然自带的alexnet模型转换报错&＃xff0c;使用说明见pytorch模型转caffe

原理呢其实很简单&＃xff0c;pytorch的model.state_dict().items()里保存了所有层的信息&＃xff0c;最朴素的方法就是将其逐个提取出来&＃xff0c;再转换为对应的caffe层&＃xff0c;pytorch转caffe步骤就是这么弄的&＃xff0c;很明显工作量很大&＃xff0c;那有没有更取巧的方法呢&＃xff1f;当然有&＃xff0c;pyTorch-To-Caffe利用了python的trace机制&＃xff0c;在回调函数中捕获网络所调用的原子操作&＃xff0c;然后将对应的操作使用caffe的python接口进行映射&＃xff0c;通过frame.f_code.co_name和frmae.f_locals可以获得网络传递过程中的函数名和参数&＃xff0c;但可惜的是完成度不高&＃xff0c;作者也没给出源码

f_code: The code object being executed in this frameco_name: Function nameco_varnames: A tuple containing the names of the local variables
f_locals: The dictionary used to look up local variables
f_back: The previous stack frame

pytorch里权重保存也是[out_channels,in_channel,h,w]的形式&＃xff0c;和caffe的是一致的&＃xff0c;拿到data直接赋值就成.

难道就没有可用的方法了吗&＃xff1f;所谓山重水复疑无路&＃xff0c;柳暗花明又一村. PytorchToCaffe作为目前完成度最高的一份代码&＃xff0c;给我们提供了很好的指南&＃xff0c;只是刚拿到这份代码时感到一头雾水&＃xff0c;不知道到底是怎么做到的.

模块初始化时会创建Rp类的对象&＃xff0c;并用这个对象覆盖pytorch中的层实现&＃xff0c;例如卷积层的实现F.conv2d&＃61;Rp(F.conv2d,_conv2d)

在工具使用中会调用pytorch网络的forward()方法&＃xff0c;此时在调用到F.conv2d层是就会调用刚才覆盖的Rp(F.conv2d,_conv2d)这个对象中的__call__方法&＃xff0c;并在此方法中调用_conv2d

_conv2d是工具内部定义的方法&＃xff0c;作用是计算pytorch中的conv&＃xff0c;将该层的名字以及计算得到的blob加入到之前创建的Translog中&＃xff0c;并创建caffe中的conv实现&＃xff0c;将pytorch中的相关权重写入caffe层中

恍然大悟&＃xff0c;其实就是用自己定义的函数替换pytorch内置的计算&＃xff0c;顺便把参数保存下来&＃xff0c;不得不说真是高明呀.

反向操作的话见把Caffe的模型转换为Pytorch模型

ResNet模块

如下的左图对应于resnet-18/34使用的基本块&＃xff0c;右图是50/101/152所使用的&＃xff0c;由于他们都比较深&＃xff0c;所以右图相比于左图使用了1x1卷积来降维。

• (a) conv3x3: 没啥好解释的&＃xff0c;将原有的pytorch函数固定卷积和尺寸为3重新封装了一次&＃xff1b;

• (b) BasicBlock: 搭建上图左边的模块。

  (1) 每个卷积块后面连接BN层进行归一化&＃xff1b;

  (2) 残差连接前的3x3卷积之后只接入BN&＃xff0c;不使用ReLU&＃xff0c;避免加和之后的特征皆为正&＃xff0c;保持特征的多样&＃xff1b;

  (3) 跳层连接&＃xff1a;两种情况&＃xff0c;当模块输入和残差支路&＃xff08;3x3->3x3&＃xff09;的通道数一致时&＃xff0c;直接相加&＃xff1b;当两者通道不一致时&＃xff08;一般发生在分辨率降低之后&＃xff0c;同分辨率一般通道数一致&＃xff09;&＃xff0c;需要对模块输入特征使用1x1卷积进行升/降维&＃xff08;步长为2&＃xff0c;上面说了分辨率会降低&＃xff09;&＃xff0c;之后同样接BN&＃xff0c;不用ReLU。

• (c)  Bottleneck: 搭建上图右边的模块。

  (1) 使用1x1卷积先降维&＃xff0c;再使用3x3卷积进行特征提取&＃xff0c;最后再使用1x1卷积把维度升回去&＃xff1b;

  (2) 每个卷积块后面连接BN层进行归一化&＃xff1b;

  (2) 残差连接前的1x1卷积之后只接入BN&＃xff0c;不使用ReLU&＃xff0c;避免加和之后的特征皆为正&＃xff0c;保持特征的多样性。

  (3) 跳层连接&＃xff1a;两种情况&＃xff0c;当模块输入和残差支路&＃xff08;1x1->3x3->1x1&＃xff09;的通道数一致时&＃xff0c;直接相加&＃xff1b;当两者通道不一致时&＃xff08;一般发生在分辨率降低之后&＃xff0c;同分辨率一般通道数一致&＃xff09;&＃xff0c;需要对模块输入特征使用1x1卷积进行升/降维&＃xff08;步长为2&＃xff0c;上面说了分辨率会降低&＃xff09;&＃xff0c;之后同样接BN&＃xff0c;不用ReLU。

def conv3x3(in_planes, out_planes, stride&＃61;1):"""3x3 convolution with padding"""return nn.Conv2d(in_planes, out_planes, kernel_size&＃61;3, stride&＃61;stride,padding&＃61;1, bias&＃61;False)class BasicBlock(nn.Module):expansion &＃61; 1def __init__(self, inplanes, planes, stride&＃61;1, downsample&＃61;None):super(BasicBlock, self).__init__()self.conv1 &＃61; conv3x3(inplanes, planes, stride)self.bn1 &＃61; nn.BatchNorm2d(planes, momentum&＃61;BN_MOMENTUM)self.relu &＃61; nn.ReLU(inplace&＃61;True)self.conv2 &＃61; conv3x3(planes, planes)self.bn2 &＃61; nn.BatchNorm2d(planes, momentum&＃61;BN_MOMENTUM)self.downsample &＃61; downsampleself.stride &＃61; stridedef forward(self, x):residual &＃61; xout &＃61; self.conv1(x)out &＃61; self.bn1(out)out &＃61; self.relu(out)out &＃61; self.conv2(out)out &＃61; self.bn2(out)if self.downsample is not None:residual &＃61; self.downsample(x)out &＃43;&＃61; residualout &＃61; self.relu(out)return outclass Bottleneck(nn.Module):expansion &＃61; 4def __init__(self, inplanes, planes, stride&＃61;1, downsample&＃61;None):super(Bottleneck, self).__init__()self.conv1 &＃61; nn.Conv2d(inplanes, planes, kernel_size&＃61;1, bias&＃61;False)self.bn1 &＃61; nn.BatchNorm2d(planes, momentum&＃61;BN_MOMENTUM)self.conv2 &＃61; nn.Conv2d(planes, planes, kernel_size&＃61;3, stride&＃61;stride,padding&＃61;1, bias&＃61;False)self.bn2 &＃61; nn.BatchNorm2d(planes, momentum&＃61;BN_MOMENTUM)self.conv3 &＃61; nn.Conv2d(planes, planes * self.expansion, kernel_size&＃61;1,bias&＃61;False)self.bn3 &＃61; nn.BatchNorm2d(planes * self.expansion,momentum&＃61;BN_MOMENTUM)self.relu &＃61; nn.ReLU(inplace&＃61;True)self.downsample &＃61; downsampleself.stride &＃61; stridedef forward(self, x):residual &＃61; xout &＃61; self.conv1(x)out &＃61; self.bn1(out)out &＃61; self.relu(out)out &＃61; self.conv2(out)out &＃61; self.bn2(out)out &＃61; self.relu(out)out &＃61; self.conv3(out)out &＃61; self.bn3(out)if self.downsample is not None:residual &＃61; self.downsample(x)out &＃43;&＃61; residualout &＃61; self.relu(out)return out