MultiModal pretraining issues

[francofbv]

After pretraining the model on WebVid, the MSRVTT evaluation results dropped to below 1%. Similarly, when pretraining from the provided pretrained weights, the results also dropped below 1% after the first step of training. We have verified our evaluation setup by using the pretrained weights and successfully replicated the results shown in the paper.

Any idea what could causing these issues?

[Andy1621]

Does the lossed seem to be normal when running?

[francofbv]

Does the lossed seem to be normal when running?

yes the loss seems fairly reasonable

[Andy1621]

It's interesting and I did not explore it before.