Error attempting to load model and resume training

[tg2k]

Using either 1.0.0.4rc4 or 1.0.0.rc5 when I save a model, reload, and call fit(), I get an exception: 'NoneType' object has no attribute 'global_local'

The error occurs because self.config_seasonality is None upon reload.

        self.num_seasonalities_modelled = len(self.id_list) if self.config_seasonality.global_local == "local" else 1

The following script triggers the problem:

import pandas as pd
from neuralprophet import NeuralProphet
import neuralprophet.utils
import numpy as np
import datetime

rows = 1000

df = pd.DataFrame({
    'ds': [datetime.datetime(2020, 1, 1) + datetime.timedelta(minutes=i) for i in range(rows)],
    'y': np.sin(2 * np.pi * np.arange(rows) / 100)
})

model = NeuralProphet(n_lags=50, n_forecasts=5, epochs=20)

metrics = model.fit(df)

neuralprophet.utils.save(model, "repro.np")

future = model.make_future_dataframe(df, periods=100)
forecast = model.predict(future)

loaded_model = neuralprophet.utils.load("repro.np")

retrain_metrics = loaded_model.fit(df)  # error occurs here

print("Done.")

FWIW, in my own (non-repro) code, it actually gets past this (it's unclear to me why), and fails downstream with unsupported operand type(s) for -: 'numpy.ndarray' and 'Timestamp' here:

arithmetic_op ([env]\lib\python3.10\site-packages\pandas\core\ops\array_ops.py:218)
_arith_method ([env]\lib\python3.10\site-packages\pandas\core\base.py:1325)
_arith_method ([env]\lib\python3.10\site-packages\pandas\core\series.py:6259)
__sub__ ([env]\lib\python3.10\site-packages\pandas\core\arraylike.py:110)
new_method ([env]\lib\python3.10\site-packages\pandas\core\ops\common.py:72)
flex_wrapper ([env]\lib\python3.10\site-packages\pandas\core\ops\__init__.py:197)
normalize ([env]\lib\python3.10\site-packages\neuralprophet\df_utils.py:416)
_normalize ([env]\lib\python3.10\site-packages\neuralprophet\data\transform.py:32)
_init_train_loader ([env]\lib\python3.10\site-packages\neuralprophet\forecaster.py:2560)
_train ([env]\lib\python3.10\site-packages\neuralprophet\forecaster.py:2642)
fit ([env]\lib\python3.10\site-packages\neuralprophet\forecaster.py:965)
...

My environment (under WSL) is stripped down to this .yml:

name: my_env

channels:
  - conda-forge
  - pytorch
  - nvidia
  - defaults
  
dependencies:
  - python=3.10

  - pip
  
  - pip:
    - neuralprophet==1.0.0rc5

[SimonWittner]

Hi @tg2k,
thanks for bringing this up - we are working on this.
Do you face the same issue, when you are using pytorch's load or save functions?

[tg2k]

@SimonWittner In the repro code, if I substitute with torch.save(model, "repro.np") and loaded_model = torch.load("repro.np"), it fails exactly the same way.

[tg2k]

I had a thought that the rows value was small enough to make this a corner case. At 1000 it is less than a day. The config_seasonality ends up set before fit() and None after fit(). This isn't to suggest that it shouldn't be addressed, but just to clarify.

Setting rows = 10000 makes the test pass the config_seasonality issue, and gets it to the unsupported operand issue. This is still with the direct calls to the Torch load/save methods.

[ourownstory]

Hi @tg2k We currently do not have support for re-fitting a fitted model - the model needs to be re-initialized. The error you are encountering is not conneced to saving and loading, but to fitting a model twice - once before saving and once after loading.

I will add a proper Error to make this clear, sorry for the confusion!

If you would like to help us support re-training / continued training, you are very welcome to do a PR changing the configuration init/storing during fitting and re-fitting.