Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deserialization of tokenizer gets loaded from huggingface #1685

Closed
whoisjones opened this issue Jun 11, 2020 · 0 comments · Fixed by #1686
Closed

Deserialization of tokenizer gets loaded from huggingface #1685

whoisjones opened this issue Jun 11, 2020 · 0 comments · Fixed by #1686
Assignees
@whoisjones
Labels
bug Something isn't working

Comments

@whoisjones
Copy link
Member

Describe the bug
At the deserialization of a model, the tokenizer gets loaded from huggingface (we're using TransformerWordEmbeddings). This throws an error when for instance using SCIBERT, because it is not supported in huggingface.

To Reproduce
Train a sequence tagger model using scibert. Then according to the docs: tagger = SequenceTagger("model_using_scibert"), which throws an errors since it is looking the scibert tokenizer in the huggingface lib.

Expected behavior
The tokenizer should not get loaded from huggingface if it is not available.

Environment (please complete the following information):

  • macOS Catalina
  • Version flair 0.5
@whoisjones whoisjones added the bug Something isn't working label Jun 11, 2020
@whoisjones whoisjones self-assigned this Jun 11, 2020
whoisjones added a commit that referenced this issue Jun 11, 2020
@whoisjones
#1685 try catch block around tokenizer.from_pretrained in order to av…
fc387be 
…oid bug
@whoisjones whoisjones mentioned this issue Jun 11, 2020
Merged
alanakbik added a commit that referenced this issue Jun 11, 2020
@alanakbik
Merge pull request #1686 from flairNLP/gh-1685-deserialization-tokeni…
79a773b 
…zer-bug

#1685 deserialization of tokenizer always gets loaded from huggingface
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@whoisjones whoisjones

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#1685 deserialization of tokenizer always gets loaded from huggingface flairNLP/flair
1 participant
@whoisjones