-
Notifications
You must be signed in to change notification settings - Fork 131
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Could not satisfy explicit device specification? #39
Comments
Apparently it something to do with embedding on GPU, I added config = tf.ConfigProto(allow_soft_placement=True) Now getting different error. Epoch 1/50 During handling of the above exception, another exception occurred: Traceback (most recent call last): Caused by op 'training/Adam/gradients/filtered_detections/map/while/TensorArrayReadV3_grad/TensorArrayGrad/TensorArrayGradV3', defined at: ...which was originally created as op 'filtered_detections/map/while/TensorArrayReadV3', defined at: NotFoundError (see above for traceback): Resource __per_step_6/_tensor_arraysfiltered_detections/map/TensorArray_0/N10tensorflow11TensorArrayE does not exist. |
I've never seen this error before, but as you also mentioned, it looks like a tensorflow error to me. Closing this for now. Feel free to reopen if you can show it is a fault of keras-maskrcnn. |
@hgaiser just a note that this error is caused when using CUDA 8 and tensorflow 1.4, Once I installed CUDA 9 and CUDA nn 7 it all worked. It would be useful for a minimum requirements section in the readme.md specifying cuda and tensorflow requirements. |
PS: Love the work. |
I see, thanks for letting us know what the error was. If you want you can make a PR to mention this requirement. |
@felipk101 i have the same error, and install CUDA 9 and CUDA nn 7 tensorflow-gpu 1.8.0 , do you have some suggestion? |
Hi, I've tried training maskrcnn on a custom dataset, and I'm getting the following crash,
It seems to come from tensorflow, I've now installed multiple versions, and tested on titan and 1070 same result.
Could it have something to do with this?
google/prettytensor#1
Running Ubuntu 16, keras 2.2.4, tensorflow 1.4
Any ideas?
/media/felix/MongoDB/Train_Test/Knox/mask5/lib/python3.5/site-packages/keras/callbacks.py:1065: UserWarning:
epsilon
argument is deprecated and will be removed, usemin_delta
instead.warnings.warn('
epsilon
argument is deprecated and '/media/felix/MongoDB/Train_Test/Knox/mask5/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py:95: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Traceback (most recent call last):
File "/media/felix/MongoDB/Train_Test/Knox/mask5/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/media/felix/MongoDB/Train_Test/Knox/mask5/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1297, in _run_fn
self._extend_graph()
File "/media/felix/MongoDB/Train_Test/Knox/mask5/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1358, in _extend_graph
self._session, graph_def.SerializeToString(), status)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/media/felix/MongoDB/Train_Test/Knox/mask5/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'training/Adam/gradients/filtered_detections/map/while/embedding_lookup_5_grad/Reshape_1/f_acc': Could not satisfy explicit device specification '' because the node was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/device:GPU:0'
Colocation Debug Info:
Colocation group had the following types and devices:
Gather: GPU CPU
ConcatV2: GPU CPU
StridedSlice: CPU
Cast: GPU CPU
TensorArrayGradV3: GPU CPU
Pack: GPU CPU
RefEnter: GPU CPU
Enter: GPU CPU
ExpandDims: GPU CPU
StackPop: GPU CPU
Stack: GPU CPU
TensorArrayReadV3: GPU CPU
Reshape: GPU CPU
UnsortedSegmentSum: GPU CPU
Identity: GPU CPU
TensorArrayGatherV3: GPU CPU
TensorArrayV3: GPU CPU
Unpack: GPU CPU
TensorArrayScatterV3: GPU CPU
Const: GPU CPU
TensorArrayWriteV3: GPU CPU
Shape: GPU CPU
Size: GPU CPU
StackPush: GPU CPU
[[Node: training/Adam/gradients/filtered_detections/map/while/embedding_lookup_5_grad/Reshape_1/f_acc = Stack_class=["loc:@filtered_detections/map/while/TensorArrayReadV3", "loc:@filtered_detections/map/while/strided_slice_10"], elem_type=DT_INT64, stack_name=""]]
The text was updated successfully, but these errors were encountered: