When fine-tuning language model, I got AttributeError: 'int' object has no attribute 'item'

[khoaipx]

Describe the bug

I fine-tuned an trained language model, but I got below error: AttributeError: 'int' object has no attribute 'item'

To Reproduce

def fine_tune(corpus_path, stored_path, model_name='mix-forward'):
    if not isinstance(stored_path, Path):
        stored_path = Path(stored_path)
    stored_path.mkdir(parents=True, exist_ok=True)

    if Path(model_name).exists():
        language_model = LanguageModel.load_language_model(model_name)
    else:
        language_model = FlairEmbeddings(model_name, fine_tune=True).lm

    # are you fine-tuning a forward or backward LM?
    is_forward_lm = language_model.is_forward_lm

    # get the dictionary from the existing language model
    dictionary: Dictionary = language_model.dictionary

    # get your corpus, process forward and at the character level
    corpus = TextCorpus(corpus_path,
                        dictionary,
                        is_forward_lm,
                        character_level=True)

    # use the model trainer to fine-tune this model on your corpus
    trainer = LanguageModelTrainer(language_model, corpus)

    trainer.train(stored_path,
                  sequence_length=100,
                  mini_batch_size=100,
                  learning_rate=20,
                  patience=10,
                  max_epochs=300,
                  checkpoint=True)

Screenshots

2019-09-17 11:47:41,099 read text file with 4 lines
2019-09-17 11:47:41,107 read text file with 4 lines
2019-09-17 11:47:41,205 read text file with 78280 lines
2019-09-17 11:47:41,265 shuffled
2019-09-17 11:48:08,470 Sequence length is 100
2019-09-17 11:48:08,502 Split 1  - (11:48:08)
2019-09-17 11:48:13,678 | split   1 /  1 |   100/  359 batches | ms/batch 51.60 | loss  1.50 | ppl     4.48
2019-09-17 11:48:18,817 | split   1 /  1 |   200/  359 batches | ms/batch 51.39 | loss  1.30 | ppl     3.67
2019-09-17 11:48:23,978 | split   1 /  1 |   300/  359 batches | ms/batch 51.60 | loss  1.24 | ppl     3.44
2019-09-17 11:48:27,022 18 seconds for train split 1
Traceback (most recent call last):
  File "main.py", line 192, in <module>
    main()
  File "main.py", line 188, in main
    fine_tune()
  File "main.py", line 180, in fine_tune
    flair_stuff.fine_tune(corpus_dir, stored_dir, model_name=model_name)
  File "/home/nolan/code/lang_model_builder/flair_stuff.py", line 314, in fine_tune
    checkpoint=True)
  File "/home/nolan/anaconda2/envs/py37/lib/python3.7/site-packages/flair/trainers/language_model_trainer.py", line 428, in train
    val_loss = self.evaluate(val_data, mini_batch_size, sequence_length)
  File "/home/nolan/anaconda2/envs/py37/lib/python3.7/site-packages/flair/trainers/language_model_trainer.py", line 511, in evaluate
    return total_loss.item() / len(data_source)
AttributeError: 'int' object has no attribute 'item'

Environment (please complete the following information):

• OS: Ubuntu 16.04
• Version: both flair 1.0 and 1.2

[alanakbik]

Hello @khoaipx - I cannot reproduce the error on my side. I've tried fine-tuning the 'news-forward-fast' model on some small text and it seems to work. Could it be an issue with your corpus?

[khoaipx]

Hi, @alanakbik, Here is my model and corpus data. Please help me to explain why my code got error!
Thank you. Have a good weekend.

[alanakbik]

Hi @khoaipx the problem is that your corpus uses a lot of characters that are not in the default dictionary, which contains only basic latin characters. You need to generate your own character dictionary by following the steps outlined here.

[khoaipx]

The existing model is trained with my own dictionary. In my code, I used the dictionary of that existing model dictionary: Dictionary = language_model.dictionary. I assume that that dictionary is my own dictionary which is built for training my model.
I also try to use my own dictionary instead of dictionary of the model with command dictionary: Dictionary = Dictionary.load_from_file(dictionary_path). But nothing changes, I also get that error. Here is my own dictionary.

[alanakbik]

Hello @khoaipx - the problem is that your validation dataset is too small - it needs to have enough characters for at least 2 mini-batches. I'll put in a PR that raises an exception to warn if the evaluation dataset is too small.

[alanakbik]

As of Flair 0.4.4, there is a new multilingual Flair model with a very large character dictionary. To fine-tune it, use this code:

from flair.embeddings import FlairEmbeddings
from flair.models import LanguageModel
from flair.trainers.language_model_trainer import LanguageModelTrainer, TextCorpus

# are you training a forward or backward LM?
is_forward_lm = True

# load the multilingual language model
language_model: LanguageModel = FlairEmbeddings('multi-forward').lm

# initialize corpus with dictionary of the multilingual language model
corpus = TextCorpus(corpus_path, language_model.dictionary, is_forward_lm, character_level=True)

# train your language model
trainer = LanguageModelTrainer(language_model, corpus)

trainer.train(
    stored_path,
    sequence_length=100,
    mini_batch_size=100,
    learning_rate=20,
    patience=10,
    max_epochs=300,
    checkpoint=True,
)