TensorFlow训练报错：ResourceExhaustedError:OOMwhenallocatingtensordevice:GPU:0byallocatorGPU_0_b

使用TensorFlow训练某些较大模型时会发生内存溢出&＃xff0c;如果 已经安装了TensorFlow-GPU版本&＃xff0c;训练时会优先调用GPU版本的TensorFlow&＃xff0c;而一般电脑上显存比较小&＃xff0c;很容易发生溢出&＃xff0c;就会出现如下报错&＃xff1a;

ResourceExhaustedError: OOM when allocating tensor with shape[1024,728,1,1] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node model/block13_sepconv2/separable_conv2d (defined at :11) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [Op:__inference_train_function_41706] Function call stack: train_function

ResourceExhaustedError: OOM when allocating tensor with shape[1024,728,1,1] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc[[node model/block13_sepconv2/separable_conv2d (defined at :11) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.[Op:__inference_train_function_41706]Function call stack:
train_function

解决方案&＃xff1a;

1. 尝试使用CPU进行训练&＃xff0c;将model.fit()代码做如下修改&＃xff1a;

with tf.device("/cpu:0"):history &＃61; model.fit(替换成自己的代码)

输出&＃xff1a;

Epoch 1/50
21/86 [&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;>.......................] - ETA: 16:08 - loss: 0.4574 - accuracy: 0.8438

这样就可以使用CPU进行训练了。

2. 如果是在jupyter notebook中运行代码&＃xff0c;则尝试先运行下面的代码&＃xff1a;

tf.keras.backend.clear_session()

如果在notebook中运行了很多代码&＃xff0c;则会占用一定的内存&＃xff0c;上面的代码顾名思义就是清楚掉之前运行的一些session&＃xff0c;以释放空间。

3. 如果还是不行&＃xff0c;则只能修改代码&＃xff0c;将批次数batch_size改小一些&＃xff0c;每次给模型喂入小批量的数据。