Cannot assign a device to node

[mschonwe]

I'm running the baby_names tutorial, and it is failing with the following error (excerpt):

tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'Adagrad/update_baby_names/embedding_lookup/params/SparseApplyAdagrad': Could not satisfy explicit device specification '' because the node was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/GPU:0'
[[Node: Adagrad/update_baby_names/embedding_lookup/params/SparseApplyAdagrad = SparseApplyAdagrad[T=DT_FLOAT, Tindices=DT_INT32, use_locking=false](baby_names/embedding_lookup/params, baby_names/embedding_lookup/params/Adagrad, ExponentialDecay, gradients/concat, gradients/concat_1)]]
Caused by op u'Adagrad/update_baby_names/embedding_lookup/params/SparseApplyAdagrad', defined at:
File "tutorial/baby_names.py", line 193, in
tf.app.run()

It was previously erroring due to the .csv not being found (so I copied into
/usr/local/lib/python2.7/dist-packages/prettytensor/tutorial/)

Any suggestions for how to fix this?

[eiderman]

Thanks for the bug report. I won't be able to reproduce this until I get to a computer with the proper version of cuda. The problem is basically that

[eiderman]

sorry, errant click.

The basic problem is that it is trying to force a non-gpu op onto a GPU. I will see where the problem is coming from.

[eiderman]

I was able to reproduce the bug and the fix was to assign a device to the problem operation:

with tf.device('/cpu:0'):
data.embedding_lookup(...)

[mschonwe]

Thanks - works now.

[rns4731]

Alternatively, you could let tensorflow pick which nodes to put it on with:
with tf.Session(config=tf.ConfigProto(
allow_soft_placement=True, log_device_placement=True)) as sess:

[abaybektursun]

@nistala Thank you. I must have missed this in the documentation.
"If you would like TensorFlow to automatically choose an existing and supported device to run the operations in case the specified one doesn't exist, you can set allow_soft_placement to True in the configuration option when creating the session."

[fmkazemi]

I encountered this problem on Tensorflow-gpu 1.8 and Tensorflow-gpu 1.5 on GPU clusters but I didn't get this issue after installing Tensorflow-gpu 1.0.1.
Also, the code below was used on all tests.
config = tf.ConfigProto()
config.gpu_options.allocator_type = 'BFC'
config.gpu_options.allow_growth = True
with tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)) as sess:

[Bella31]

Encountered a similar problem with Tensorflow-gpu 1.8 with the function save/SaveV2.
I get the error "Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available." even though I specify in my session:
config=tf.ConfigProto(allow_soft_placement=True)

[lucaswu]

@Bella31 , have you fixed this problem. I have the same problem. Thanks.

[Bella31]

No, I didn't, also will be happy to hear if there is a solution.

[Bella31]

Actually, the problem was in my code. It turned out that I called Saver in "with tf.device(device)" clause. When I put it under "with tf.Session(config=tf.ConfigProto(allow_soft_placement=True))" it was resolved

[RayXu14]

I think giving using InteractiveSession might help. It seems that use Session and that operation can be done on gpu.