CUDA_ERROR_OUT_OF_MEMORY in tensorflow

When I started to train some neural network, it met the CUDA_ERROR_OUT_OF_MEMORY but the training could go on without error. Because I wanted to use gpu memory as it really needs, so I set the gpu_options.allow_growth = True.The logs are as follows:

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.7335
pciBusID 0000:01:00.0
Total memory: 7.92GiB
Free memory: 7.81GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device:0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0)
E tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
Iter 20, Minibatch Loss= 40491.636719
...

And after using nvidia-smi command, it gets:

+-----------------------------------------------------------------------------+   
| NVIDIA-SMI 367.27                 Driver Version: 367.27                            
|-------------------------------+----------------------+----------------------+   
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |  
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M.
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 0000:01:00.0     Off |                  N/A |   
| 40%   61C    P2    46W / 180W |   8107MiB /  8111MiB |     96%      Default |   
+-------------------------------+----------------------+----------------------+   
|   1  GeForce GTX 1080    Off  | 0000:02:00.0     Off |                  N/A |   
|  0%   40C    P0    40W / 180W |      0MiB /  8113MiB |      0%      Default |   
+-------------------------------+----------------------+----------------------+   
                                                                              │
+-----------------------------------------------------------------------------+   
| Processes:                                                       GPU Memory |   
|  GPU       PID  Type  Process name                               Usage      |   
|=============================================================================|   
|    0     22932    C   python                                        8105MiB |
+-----------------------------------------------------------------------------+

After I commented the gpu_options.allow_growth = True, I trained the net again and everything was normal. There was no the problem of CUDA_ERROR_OUT_OF_MEMORY. Finally, ran the nvidia-smi command, it gets:

+-----------------------------------------------------------------------------+   
| NVIDIA-SMI 367.27                 Driver Version: 367.27                            
|-------------------------------+----------------------+----------------------+   
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |  
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M.
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 0000:01:00.0     Off |                  N/A |   
| 40%   61C    P2    46W / 180W |   7793MiB /  8111MiB |     99%      Default |   
+-------------------------------+----------------------+----------------------+   
|   1  GeForce GTX 1080    Off  | 0000:02:00.0     Off |                  N/A |   
|  0%   40C    P0    40W / 180W |      0MiB /  8113MiB |      0%      Default |   
+-------------------------------+----------------------+----------------------+   
                                                                              │
+-----------------------------------------------------------------------------+   
| Processes:                                                       GPU Memory |   
|  GPU       PID  Type  Process name                               Usage      |   
|=============================================================================|   
|    0     22932    C   python                                        7791MiB |
+-----------------------------------------------------------------------------+

I have two questions about it. Why did the CUDA_OUT_OF_MEMORY come out and the procedure went on normally? why did the memory usage become smaller after commenting allow_growth = True.

Related Posts:

  1. What is the difference between model.fit() an model.evaluate() in Keras?
  2. ImportError: cannot import name ‘abs’
  3. Could not install packages due to an EnvironmentError: [WinError 5] Access is denied:
  4. Tensorflow can’t find “cudart64_90.dll” even though it is installed with Path variable set
  5. Understanding the ResourceExhaustedError: OOM when allocating tensor with shape
  6. Is there a tensorflow version that is compatible with Cuda 9.0 and cudnn 7.1
  7. Tensorflow: When should I use or not use `feed_dict`?
  8. Tensorflow: ImportError: libcusolver.so.8.0: cannot open shared object file: No such file or directory
  9. Does TensorFlow plan to support OpenCL?
  10. Tensorflow import error: No module named ‘tensorflow’
  11. ImportError: No module named tensorflow
  12. Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
  13. Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
  14. How to find which version of TensorFlow is installed in my system?
  15. Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
  16. ImportError: No module named ‘tensorflow.python’
  17. What is the meaning of the word logits in TensorFlow?
  18. Could not find a version that satisfies the requirement tensorflow
  19. Could not find a version that satisfies the requirement tensorflow
  20. ImportError: No module named ‘keras’
  21. ImportError: libcublas.so.9.0: cannot open shared object file
  22. Convert a tensor to numpy array in Tensorflow?
  23. Convert a tensor to numpy array in Tensorflow?
  24. how to find libstdc++.so.6: that contain GLIBCXX_3.4.19 for RHEL 6?
  25. Convert a tensor to numpy array in Tensorflow?
  26. Tensorflow: why ‘pip uninstall tensorflow’ cannot find tensorflow
  27. Installing tensorflow with anaconda in windows
  28. Cannot uninstall Tensorflow
  29. error: Failed to load the native TensorFlow runtime
  30. Can I run Keras model on gpu?
  31. What is the difference between sparse_categorical_crossentropy and categorical_crossentropy?
  32. How to tell if tensorflow is using gpu acceleration from inside python shell?
  33. pip3: command not found
  34. ImportError: libcuda.so.1: cannot open shared object file
  35. Import error when trying to import tensorflow with gpu
  36. LSTM Keras input shape confusion
  37. What is the difference between np.mean and tf.reduce_mean?
  38. Failed to load the native TensorFlow runtime – TensorFlow 2.1
  39. No Module Named ‘_pywrap_tensorflow_internal’
  40. Tensorflow not running on GPU
  41. Uninstalling TensorFlow from Anaconda environment
  42. TensorFlow: Blas GEMM launch failed
  43. What’s the difference between scikit-learn and tensorflow? Is it possible to use them together?
  44. Error: OOM when allocating tensor with shape
  45. tar: Unrecognized archive format error when trying to unpack flower_photos.tgz, TF tutorials on OSX
  46. can’t open tensorboard 0.0.0.0:6006 or localhost:6006
  47. How to fix “AttributeError: module ‘tensorflow’ has no attribute ‘get_default_graph'”?
  48. TensorFlow not found using pip
  49. Tensorflow 2.0 – AttributeError: module ‘tensorflow’ has no attribute ‘Session’
  50. No module named ‘tqdm’
  51. What is the difference between ‘SAME’ and ‘VALID’ padding in tf.nn.max_pool of tensorflow?
  52. RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
  53. Install pip for python 3.5
  54. How to change python version in Anaconda?
  55. AttributeError: ‘module’ object has no attribute ‘main’ for tf.app.run()
  56. How to downgrade tensorflow, multiple versions possible?
  57. TensorfloTensorflow installation error: not a supported wheel on this platformw installation error: not a supported wheel on this platform
  58. TensorFlow: how is dataset.train.next_batch defined?

Leave a Comment