-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
en-sentiment model contains labels that don't match IMDB dataset #1165
Comments
FYI I ran the Imdb model against some opinion based news and I didn't find the results to be all that meaningful. |
Yes this model was trained using IMDB data, i.e. film related, so its only a sentiment model for movie reviews. I have to check why the label names changed, very strange! |
@alanakbik This is very strange. from flair.datasets import IMDB, DataLoader
from flair.models import TextClassifier
classifier = TextClassifier.load('en-sentiment')
corpus = IMDB()
sentences = list(corpus.test) It downloads what it has to.
Then I do sentences[0].labels
# Out[12]: [pos (1.0)] And a = classifier.predict(sentences=sentences[:100],
mini_batch_size=16,
embedding_storage_mode="none")
a[0].labels
# Out[13]: [POSITIVE (0.9999998807907104)] So labels ARE different. Of course, now: sentences[0].labels
# Out[14]: [POSITIVE (0.9999998807907104)] as sentence labels are overriden by first line of
To make it short, label from model is |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
PR #1545 adds new sentiment datasets and homogenizes labels across datasets. Also, there is now an option to define "name maps" to map label names to other names. |
Describe the bug
According to TUTORIAL 2, classification model en-sentiment has been trained on IMDB dataset.
evaluate()
function on en-sentiment model trained on IMDB produces 0 score when tested on test set of IMDB.Reasons seem to be a change in label names.
Model: ['POSITIVE'], ['NEGATIVE']
dataset: ['pos'], ???
I have not a single prediction with negative label. (which is another issue)
IMDB is not marked as deprecated in source code.
To Reproduce
Print:
Expected behavior
A score which is not zero
Solution
Retrain / reshare a new en-sentiment model? // redo IMDB dataset
Context
PiPy version of Flair / same bug on master branch
Note
Btw, the model seems to not work very well.
"I never saw something that bad." -> positive
"I do not like this film" -> positive
"I hate this film" -> negative (finally)
The text was updated successfully, but these errors were encountered: