RoBERTa model

[stefan-it]

Hi,

thanks for releasing the RoBERTa model ❤️

I've one question regarding to the output features:

features = roberta.extract_features(tokens)
features.size()
torch.Size([1, 5, 1024])

Are these features the output of the last layer (layer nr. 24) of the Transformer model? Is it currently possible to select a specific layer?

Thanks in advance,

Stefan

[myleott]

Yes, it’s the last layer before lm_head. I will add an option to expose specific layers, but for now you can copy this: https://github.com/pytorch/fairseq/blob/master/fairseq/models/roberta.py#L106

If you add return_all_hiddens=True, then the second element of the returned tuple will contain all of the inner states.

[myleott]

Exposed new functionality in #909:

>>> roberta.eval()

>>> tokens = roberta.encode('Hello world.')

>>> last_layer_features = roberta.extract_features(tokens)
>>> last_layer_features.size()
torch.Size([1, 6, 1024])

>>> all_layers = roberta.extract_features(tokens, return_all_hiddens=True)
>>> len(all_layers)
25
>>> all_layers[-1].size()
torch.Size([1, 6, 1024])

>>> torch.all(all_layers[-1] == last_layer_features)
tensor(1, dtype=torch.uint8)

[stefan-it]

Hi @myleott thanks for your kind help and extending the interface ❤️ I could now integrate the RoBERTa into an upcoming version of Flair 🤗