I’m trying to implement a skip thought model using tensorflow and a current version is placed here.
Currently I using one GPU of my machine (total 2 GPUs) and the GPU info is
2017-09-06 11:29:32.657299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate (GHz) 1.683 pciBusID 0000:02:00.0 Total memory: 10.91GiB Free memory: 10.75GiB
However, I got OOM when I’m trying to feed data to the model. I try to debug as follow:
I use the following snippet right after I run sess.run(tf.global_variables_initializer())
logger.info('Total: {} params'.format(
np.sum([
np.prod(v.get_shape().as_list())
for v in tf.trainable_variables()
])))
and got 2017-09-06 11:29:51,333 INFO main main.py:127 - Total: 62968629 params, roughly about 240Mb if all using tf.float32. The output of tf.global_variables is
[<tf.Variable 'embedding/embedding_matrix:0' shape=(155229, 200) dtype=float32_ref>, <tf.Variable 'encoder/rnn/gru_cell/gates/kernel:0' shape=(400, 400) dtype=float32_ref>, <tf.Variable 'encoder/rnn/gru_cell/gates/bias:0' shape=(400,) dtype=float32_ref>, <tf.Variable 'encoder/rnn/gru_cell/candidate/kernel:0' shape=(400, 200) dtype=float32_ref>, <tf.Variable 'encoder/rnn/gru_cell/candidate/bias:0' shape=(200,) dtype=float32_ref>, <tf.Variable 'decoder/weights:0' shape=(200, 155229) dtype=float32_ref>, <tf.Variable 'decoder/biases:0' shape=(155229,) dtype=float32_ref>, <tf.Variable 'decoder/previous_decoder/rnn/gru_cell/gates/kernel:0' shape=(400, 400) dtype=float32_ref>, <tf.Variable 'decoder/previous_decoder/rnn/gru_cell/gates/bias:0' shape=(400,) dtype=float32_ref>, <tf.Variable 'decoder/previous_decoder/rnn/gru_cell/candidate/kernel:0' shape=(400, 200) dtype=float32_ref>, <tf.Variable 'decoder/previous_decoder/rnn/gru_cell/candidate/bias:0' shape=(200,) dtype=float32_ref>, <tf.Variable 'decoder/next_decoder/rnn/gru_cell/gates/kernel:0' shape=(400, 400) dtype=float32_ref>, <tf.Variable 'decoder/next_decoder/rnn/gru_cell/gates/bias:0' shape=(400,) dtype=float32_ref>, <tf.Variable 'decoder/next_decoder/rnn/gru_cell/candidate/kernel:0' shape=(400, 200) dtype=float32_ref>, <tf.Variable 'decoder/next_decoder/rnn/gru_cell/candidate/bias:0' shape=(200,) dtype=float32_ref>, <tf.Variable 'global_step:0' shape=() dtype=int32_ref>]
In my training phrase, I have a data array whose shape is (164652, 3, 30), namely sample_size x 3 x time_step, the 3 here means the previous sentence, current sentence and next sentence. The size of this training data is about 57Mb and is stored in a loader. Then I use write a generator function to get the sentences, looks like
def iter_batches(self, batch_size=128, time_major=True, shuffle=True):
num_samples = len(self._sentences)
if shuffle:
samples = self._sentences[np.random.permutation(num_samples)]
else:
samples = self._sentences
batch_start = 0
while batch_start < num_samples:
batch = samples[batch_start:batch_start + batch_size]
lens = (batch != self._vocab[self._vocab.pad_token]).sum(axis=2)
y, x, z = batch[:, 0, :], batch[:, 1, :], batch[:, 2, :]
if time_major:
yield (y.T, lens[:, 0]), (x.T, lens[:, 1]), (z.T, lens[:, 2])
else:
yield (y, lens[:, 0]), (x, lens[:, 1]), (z, lens[:, 2])
batch_start += batch_size
The training loop looks like
for epoch in num_epochs:
batches = loader.iter_batches(batch_size=args.batch_size)
try:
(y, y_lens), (x, x_lens), (z, z_lens) = next(batches)
_, summaries, loss_val = sess.run(
[train_op, train_summary_op, st.loss],
feed_dict={
st.inputs: x,
st.sequence_length: x_lens,
st.previous_targets: y,
st.previous_target_lengths: y_lens,
st.next_targets: z,
st.next_target_lengths: z_lens
})
except StopIteraton:
...
Then I got a OOM. If I comment out the whole try body (no to feed data), the script run just fine.
I have no idea why I got OOM in such a small data scale. Using nvidia-smi I always got
Wed Sep 6 12:03:37 2017 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 384.59 Driver Version: 384.59 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce GTX 108... Off | 00000000:02:00.0 Off | N/A | | 0% 44C P2 60W / 275W | 10623MiB / 11172MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 GeForce GTX 108... Off | 00000000:03:00.0 Off | N/A | | 0% 43C P2 62W / 275W | 10621MiB / 11171MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 32748 C python3 10613MiB | | 1 32748 C python3 10611MiB | +-----------------------------------------------------------------------------+
I can’t see the actual GPU usage of my script since tensorflow always steals all memory at the beginning. And the actual problem here is I don’t know how to debug this.
I’ve read some posts about OOM on StackOverflow. Most of them happened when feeding a large test set data to the model and feeding the data by small batches can avoid the problem. But I don’t why see such a small data and param combination sucks in my 11Gb 1080Ti, since the error it just try to allocate a matrix sized [3840 x 155229]. (The output matrix of the decoder, 3840 = 30(time_steps) x 128(batch_size), 155229 is vocab_size).
2017-09-06 12:14:45.787566: W tensorflow/core/common_runtime/bfc_allocator.cc:277] ********************************************************************************************xxxxxxxx
2017-09-06 12:14:45.787597: W tensorflow/core/framework/op_kernel.cc:1158] Resource exhausted: OOM when allocating tensor with shape[3840,155229]
2017-09-06 12:14:45.788735: W tensorflow/core/framework/op_kernel.cc:1158] Resource exhausted: OOM when allocating tensor with shape[3840,155229]
[[Node: decoder/previous_decoder/Add = Add[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](decoder/previous_decoder/MatMul, decoder/biases/read)]]
2017-09-06 12:14:45.790453: I tensorflow/core/common_runtime/gpu/pool_allocator.cc:247] PoolAllocator: After 2857 get requests, put_count=2078 evicted_count=1000 eviction_rate=0.481232 and unsatisfied allocation rate=0.657683
2017-09-06 12:14:45.790482: I tensorflow/core/common_runtime/gpu/pool_allocator.cc:259] Raising pool_size_limit_ from 100 to 110
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1139, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1121, in _run_fn
status, run_metadata)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3840,155229]
[[Node: decoder/previous_decoder/Add = Add[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](decoder/previous_decoder/MatMul, decoder/biases/read)]]
[[Node: GradientDescent/update/_146 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_2166_GradientDescent/update", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
During handling of the above exception, another exception occurred:
Any help will be appreciated. Thanks in advance.