[BUG] - Error when using tf.data.Dataset on metrics

[Agustin-Picard]

Select the modules to which the bug refers:

• Metrics

Describe the bug
Passing a tf.data.Dataset to metrics constructors derives into the program incorrectly accessing an element_spec of the aforementioned dataset as if it were a list when it is not the case.

Screenshots

2022-03-17 11:44:57.703227: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2022-03-17 11:44:57.704477: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-03-17 11:45:01.816089: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-03-17 11:45:01.817565: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2022-03-17 11:45:01.817889: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2022-03-17 11:45:01.822735: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: B201801349
2022-03-17 11:45:01.823202: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: B201801349
2022-03-17 11:45:01.824379: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-03-17 11:45:01.826257: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
WARNING:tensorflow:Model was constructed with shape (1, 32, 32, 3) for input KerasTensor(type_spec=TensorSpec(shape=(1, 32, 32, 3), dtype=tf.float32, name='conv2d_input'), name='conv2d_input', description="created by layer 'conv2d_input'"), but it was called on an input with incompatible shape (64, 32, 32, 3).
2022-03-17 11:45:03.985713: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
WARNING:tensorflow:Model was constructed with shape (1, 32, 32, 3) for input KerasTensor(type_spec=TensorSpec(shape=(1, 32, 32, 3), dtype=tf.float32, name='conv2d_input'), name='conv2d_input', description="created by layer 'conv2d_input'"), but it was called on an input with incompatible shape (56, 32, 32, 3).
WARNING:tensorflow:AutoGraph could not transform <function <lambda> at 0x000001D6E9D67D30> and will run it as-is.
Cause: could not parse the source code of <function <lambda> at 0x000001D6E9D67D30>: found multiple definitions with identical signatures at the location. This error may be avoided by defining each lambda on a single line and with unique argument names.
Match 0:
(lambda z, w: z)

Match 1:
(lambda z, w: w)

To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <function <lambda> at 0x000001D6E9D67E50> and will run it as-is.
Cause: could not parse the source code of <function <lambda> at 0x000001D6E9D67E50>: found multiple definitions with identical signatures at the location. This error may be avoided by defining each lambda on a single line and with unique argument names.
Match 0:
(lambda z, w: z)

Match 1:
(lambda z, w: w)

To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
Traceback (most recent call last):
  File "C:/Users/a-m.picard/AppData/Roaming/JetBrains/PyCharmCE2021.3/scratches/scratch_1.py", line 24, in <module>
    metric = Deletion(model, dataset.map(lambda z, w: z), dataset.map(lambda z, w: w))
  File "C:\Users\a-m.picard\Anaconda3\envs\scouter-tf2\lib\site-packages\xplique\metrics\fidelity.py", line 324, in __init__
    super().__init__(model, inputs, targets, batch_size, "deletion",
  File "C:\Users\a-m.picard\Anaconda3\envs\scouter-tf2\lib\site-packages\xplique\metrics\fidelity.py", line 174, in __init__
    super().__init__(model, inputs, targets, batch_size)
  File "C:\Users\a-m.picard\Anaconda3\envs\scouter-tf2\lib\site-packages\xplique\metrics\base.py", line 36, in __init__
    self.inputs, self.targets = numpy_sanitize(inputs, targets)
  File "C:\Users\a-m.picard\Anaconda3\envs\scouter-tf2\lib\site-packages\xplique\commons\data_conversion.py", line 69, in numpy_sanitize
    inputs, targets = tensor_sanitize(inputs, targets)
  File "C:\Users\a-m.picard\Anaconda3\envs\scouter-tf2\lib\site-packages\xplique\commons\data_conversion.py", line 35, in tensor_sanitize
    dataset_shape = inputs.element_spec[0].shape
TypeError: 'TensorSpec' object is not subscriptable

Desktop (please complete the following information):

• OS: Windows, Linux
• Python version: 3.9.5
• Tensorflow version: 2.4.1 - 2.7

To Reproduce

import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Conv2D, GlobalAveragePooling2D, Dense
from xplique.attributions import Saliency
from xplique.metrics import Deletion

x = tf.random.normal((120, 32, 32, 3))
y = tf.random.normal((120, 10))
dataset = tf.data.Dataset.from_tensor_slices((x, y))
model = Sequential([
    Conv2D(8, 3, strides=2, padding='same'),
    Conv2D(8, 3, strides=2, padding='same'),
    GlobalAveragePooling2D(),
    Dense(10)
])
model(tf.random.normal((1, 32, 32, 3)))
explainer = Saliency(model, -1, batch_size=64)
explanations = []
for batch_x, batch_y in dataset.batch(64):
    exp = explainer(batch_x, batch_y)
    explanations.extend(exp)
metric = Deletion(model, dataset.map(lambda z, w: z), dataset.map(lambda z, w: w))
metric_score = metric(explanations)

Expected behavior
I would expect the metric to be computed, but whilst instantiating the Deletion object, the constructor attempts to figure out the shapes of the inputs of the individual points of a tf.data.Dataset. Whilst typically straight-forward, a simple subscript bug appears, probably stemming from a incorrectly treated edge case.

Additional context
The property element_spec of a dataset is not a list, and thus, the subscript is unnecessary for the desired function.

[adubowski]

Any progress on that? I am facing the same issue

[fel-thomas]

Thanks for your comment @adubowski, @Agustin-Picard !
The next Xplique version should be deploy on pip in the next hour, and It should be working now (addressed in #122).

Don't forget to close the issue if it works for you :)

[adubowski]

Hi @fel-thomas, thank you for looking into it! I am afraid it is not completely fixed yet though, as the metrics' __init__ still checks for shape. In my case, the issue is raise in line 178 of fidelity.py (stacktrace below)

    315 def __init__(self,
    316              model: tf.keras.Model,
    317              inputs: Union[tf.data.Dataset, tf.Tensor, np.ndarray],
   (...)
    322              max_percentage_perturbed: float = 1.0
    323              ):
--> 324     super().__init__(model, inputs, targets, batch_size, "deletion",
    325                      baseline_mode, steps, max_percentage_perturbed)

File ~\.virtualenvs\paif-explanations-NwzKkAFY\lib\site-packages\xplique\metrics\fidelity.py:178, in CausalFidelity.__init__(self, model, inputs, targets, batch_size, causal_mode, baseline_mode, steps, max_percentage_perturbed)
    175 self.causal_mode = causal_mode
    176 self.baseline_mode = baseline_mode
--> 178 self.nb_features = np.prod(inputs.shape[1:-1])
    179 self.inputs_flatten = inputs.reshape((len(inputs), self.nb_features, inputs.shape[-1]))
    181 assert 0.0 < max_percentage_perturbed <= 1.0, "`max_percentage_perturbed` must be" \
    182                                               "in ]0, 1]."

AttributeError: 'ShuffleDataset' object has no attribute 'shape' ```