Uncertainty: Conformal Prediction V1.1 - extend to multiple forecast steps instead of only a single forecast step #1073

Kevin-Chen0 · 2022-12-17T06:58:41Z

🔬 Background

Resolves the multiple forecast steps task found in Issue Conformal Prediction V1.1 tasks #1017.

🔮 Key changes

Extend to multiple forecast steps instead of only a single forecast step.

📋 Review Checklist

I have performed a self-review of my own code.
I have commented my code, added docstrings and data types to function definitions.
I have added pytests to check whether my feature / fix works.

Please make sure to follow our best practices in the Contributing guidelines.

codecov-commenter · 2022-12-17T07:03:30Z

Codecov Report

Merging #1073 (f4629e2) into main (05d5e10) will increase coverage by 0.19%.
The diff coverage is 97.91%.

@@            Coverage Diff             @@
##             main    #1073      +/-   ##
==========================================
+ Coverage   90.14%   90.33%   +0.19%     
==========================================
  Files          21       21              
  Lines        4800     4824      +24     
==========================================
+ Hits         4327     4358      +31     
+ Misses        473      466       -7

Impacted Files	Coverage Δ
neuralprophet/conformal.py	`96.82% <96.96%> (+2.59%)`	⬆️
neuralprophet/forecaster.py	`87.89% <100.00%> (ø)`
neuralprophet/plot_forecast_matplotlib.py	`85.50% <100.00%> (+0.58%)`	⬆️
neuralprophet/plot_forecast_plotly.py	`87.72% <100.00%> (+3.07%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2022-12-17T07:03:35Z

Model Benchmark

Benchmark	Metric	main	current	diff
AirPassengers	MAE_val	15.2698	15.2698	0.0%	✅
AirPassengers	RMSE_val	19.4209	19.4209	0.0%	✅
AirPassengers	Loss_val	0.00195	0.00195	0.0%	✅
AirPassengers	MAE	9.82902	9.82902	0.0%	✅
AirPassengers	RMSE	11.7005	11.7005	0.0%	✅
AirPassengers	Loss	0.00056	0.00056	0.0%	✅
AirPassengers	time	4.90997	4.08	-16.9%	🎉
YosemiteTemps	MAE_val	1.72948	1.72949	0.0%	✅
YosemiteTemps	RMSE_val	2.27386	2.27386	0.0%	✅
YosemiteTemps	Loss_val	0.00096	0.00096	0.0%	✅
YosemiteTemps	MAE	1.45189	1.45189	0.0%	✅
YosemiteTemps	RMSE	2.16631	2.16631	0.0%	✅
YosemiteTemps	Loss	0.00066	0.00066	0.0%	✅
YosemiteTemps	time	111.253	92.43	-16.92%	🎉
PeytonManning	MAE_val	0.64636	0.64636	0.0%	✅
PeytonManning	RMSE_val	0.79276	0.79276	0.0%	✅
PeytonManning	Loss_val	0.01494	0.01494	0.0%	✅
PeytonManning	MAE	0.42701	0.42701	0.0%	✅
PeytonManning	RMSE	0.57032	0.57032	0.0%	✅
PeytonManning	Loss	0.00635	0.00635	0.0%	✅
PeytonManning	time	13.5751	11.42	-15.88%	🎉

Model training plots

Model Training

PeytonManning

YosemiteTemps

AirPassengers

…nd added plot_interval_width_per_timestep() method for multiple timesteps.

…ion.py.

…n and changed the split frequencies from to because of the Peyton Manning dataset.

…ltistep

Kevin-Chen0 · 2023-01-05T06:01:42Z

FYI, I had to comment out test_PeytonManning, test_YosemiteTemps, and test_AirPassengers in test_model_performance.py as they were failing for me.

…in test_plotting.py.

…hod in conformal.py.

ourownstory · 2023-01-13T00:27:57Z

neuralprophet/conformal.py

+                    f"Unknown conformal prediction method '{self.method}'. Please input either 'naive' or 'cqr'."
+                )
+            if step_number == 1:
+                # save nonconformity scores of the first timestep


Why are these saved for the first step (only)?

The nonconformity scores are saved in order to be inputted into plot_nonconformity_scores() , where it plots:

The nonconformity score is the score (blue line).

This plot is only called when n_forecasts ==1. For models with n_forecasts >1, it will call the plot_interval_width_per_timestep() instead (more details under your 3rd question), which doesn't require nonconformity scores. Therefore, only the first step needs to be saved.

Thank you for the explanation!

ourownstory · 2023-01-13T00:33:55Z

neuralprophet/conformal.py

-            fig = plot_nonconformity_scores(self.noncon_scores, self.alpha, self.q_hat, method)
+            if self.n_forecasts == 1:
+                # includes nonconformity scores of the first timestep
+                fig = plot_nonconformity_scores(self.noncon_scores, self.alpha, self.q_hats[0], method)


Ideally we would have the same behavior for self.n_forecasts == 1 and other values of n_forecasts.
Maybe we could instead have the special plot that includes the scores be a separate plotting utility call instead of automatically overwriting the standard plotting utility.

Further, there seems to be these two options only in the matplotlib case but not for plotly - However the plot should be as identical as possible independent of the plotting backend. How can we make them return the same kind of plot?

Two different plots show whether n_forecasts == 1 or n_forecasts >1. See your 1st and 3rd questions for details.

As for the matplotlib and plotly, both have the plot_nonconformity_scores(), but only matplotlib has the plot_interval_width_per_timestep(). I'll try to code that for plotly now and before the merge of this PR.

Just added plot_interval_width_per_timestep() for for plotly and modified the test_plot_conformal_prediction so the test coverage for that addition is there.

It makes more sense to me now, thank you!
From a UI-perspective it's not ideal to have the plot change quite so drastically depending on an external factor, but I think it is OK in this case as this is more of a research/diagnostic plot.

ourownstory · 2023-01-13T00:41:05Z

neuralprophet/plot_forecast_matplotlib.py

+            Figure showing the q-values for each timestep
+    """
+    fig, ax = plt.subplots()
+    ax.plot(range(1, len(q_hats) + 1), q_hats)


I am sorry, I am a bit confused about what is being plotted based on the docstring and code.
Aren't the qhats static?
Or what are we plotting here?

This is the plot_interval_width_per_timestep() method for models with n_lags > 1 and thus self.n_forecasts >1. You can try with the m3 and m4 models in the uncertainty_conformal_prediction.ipynb. Therefore, instead of plotting the Naive One-Side Interval Width from q (with the noncon scores and q1 horizontal line from just forecast1), this method will instead plot the q values (or q_hats) for each of the timesteps:

So the blue line here is not the noncon scores but the q values on each timestep number.

You can see how q linearly increases the further the timestep (until it plateaus around t+12)? This shows the greater uncertainty, the further out the forecast that it requires wider intervals for the same confidence level (e.g., 90% in this example). The reason for this plateau could be attributed to the half-day seasonality of this hospital energy load dataset (i.e., day and night usage).

Thank you for explaining!
Sorry, I interpreted x-axis to be datetime, not forecast step numbers.
Now it makes sense. Cool how one can see it increase and then plateau!
I like this new plot. I think it is helpful!

…nd modified plot() in conformal.py to enable this method for plotting_backend='plotly'.

… plotting_backend param into m.conformal_predict() method.

ourownstory

Thank you for adding the plotly equivalent! Good work!
Ready to merge.

Replaced yhat1 with the step_number for conformal.

fc1936a

Kevin-Chen0 added enhancement status: needs review PR needs to be reviewed by Reviewer(s) priority:P2 Medium priority labels Dec 17, 2022

Kevin-Chen0 added this to the Release 0.5.1 milestone Dec 17, 2022

Kevin-Chen0 requested a review from ourownstory December 17, 2022 06:58

Kevin-Chen0 self-assigned this Dec 17, 2022

Moved conformal_predict() method logic into conformal_prediction.py a…

322b239

…nd added plot_interval_width_per_timestep() method for multiple timesteps.

Kevin-Chen0 added status: needs update PR has outstanding comment(s) or PR test(s) that need to be resolved and removed status: needs review PR needs to be reviewed by Reviewer(s) labels Dec 18, 2022

Kevin-Chen2 and others added 9 commits December 18, 2022 06:24

Changed self.config_train.quantiles to quantiles in conformal_predict…

e084a41

…ion.py.

Removed q_hats in _conformalize().

d2be041

Merge branch 'main' into refactor/conformal-multistep

e366bb6

Merge branch 'main' into refactor/conformal-multistep

d6b229a

Added Conformal dataclass from PR #1073.

bd4b225

Added conformal.py to replace conformal_prediction.py.

feef1d2

Fixed self.method in ValueError for conformal.py.

fa5e861

Modified Conformal class to fit multiple lines in forecaster.py.

54458bf

Uncommented auto-regression section for test_plot_conformal_predictio…

ed989ff

…n and changed the split frequencies from to because of the Peyton Manning dataset.

Kevin-Chen0 requested a review from noxan January 3, 2023 23:40

Kevin-Chen0 added status: needs review PR needs to be reviewed by Reviewer(s) and removed status: needs update PR has outstanding comment(s) or PR test(s) that need to be resolved labels Jan 3, 2023

Kevin-Chen0 and others added 2 commits January 3, 2023 18:45

Merge branch 'main' into refactor/conformal-multistep

d2863a7

Merge remote-tracking branch 'origin/main' into refactor/conformal-mu…

889a8fa

…ltistep

Kevin-Chen0 and others added 2 commits January 8, 2023 11:29

Merge branch 'main' into refactor/conformal-multistep

4fbe1b5

Uncommented m.plot_latest_forecast in test_plot_conformal_prediction …

195fe60

…in test_plotting.py.

Kevin-Chen0 and others added 3 commits January 9, 2023 18:22

Merge branch 'main' into refactor/conformal-multistep

9c91965

Added step_number in docstring in the _get_nonconformity_scores() met…

d491bea

…hod in conformal.py.

Uncommented the tests in test_model_performance.py.

8ad5088

Kevin-Chen0 linked an issue Jan 10, 2023 that may be closed by this pull request

Conformal Prediction V1.1 tasks #1017

Closed

3 tasks

Merge branch 'main' into refactor/conformal-multistep

1a9e165

ourownstory reviewed Jan 13, 2023

View reviewed changes

Kevin-Chen0 and others added 3 commits January 12, 2023 20:42

Merge branch 'main' into refactor/conformal-multistep

2aebf76

Added plot_interval_width_per_timestep() to plot_forecast_plotly.py a…

2a1ddf4

…nd modified plot() in conformal.py to enable this method for plotting_backend='plotly'.

Modified test_plot_conformal_prediction in test_plotting.py by adding…

f4629e2

… plotting_backend param into m.conformal_predict() method.

ourownstory approved these changes Jan 13, 2023

View reviewed changes

ourownstory merged commit 3d46b1e into main Jan 13, 2023

ourownstory deleted the refactor/conformal-multistep branch January 13, 2023 07:40

Kevin-Chen0 restored the refactor/conformal-multistep branch February 20, 2023 21:20

Kevin-Chen0 deleted the refactor/conformal-multistep branch February 20, 2023 21:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uncertainty: Conformal Prediction V1.1 - extend to multiple forecast steps instead of only a single forecast step #1073

Uncertainty: Conformal Prediction V1.1 - extend to multiple forecast steps instead of only a single forecast step #1073

Kevin-Chen0 commented Dec 17, 2022 •

edited

Loading

codecov-commenter commented Dec 17, 2022 •

edited

Loading

github-actions bot commented Dec 17, 2022 •

edited

Loading

Model Training

PeytonManning

YosemiteTemps

AirPassengers

Kevin-Chen0 commented Jan 5, 2023

ourownstory Jan 13, 2023

Kevin-Chen0 Jan 13, 2023

ourownstory Jan 13, 2023 •

edited

Loading

ourownstory Jan 13, 2023

ourownstory Jan 13, 2023

Kevin-Chen0 Jan 13, 2023

Kevin-Chen0 Jan 13, 2023

ourownstory Jan 13, 2023

ourownstory Jan 13, 2023

Kevin-Chen0 Jan 13, 2023 •

edited

Loading

ourownstory Jan 13, 2023 •

edited

Loading

ourownstory left a comment

Uncertainty: Conformal Prediction V1.1 - extend to multiple forecast steps instead of only a single forecast step #1073

Uncertainty: Conformal Prediction V1.1 - extend to multiple forecast steps instead of only a single forecast step #1073

Conversation

Kevin-Chen0 commented Dec 17, 2022 • edited Loading

🔬 Background

🔮 Key changes

📋 Review Checklist

codecov-commenter commented Dec 17, 2022 • edited Loading

Codecov Report

github-actions bot commented Dec 17, 2022 • edited Loading

Model Benchmark

Model Training

PeytonManning

YosemiteTemps

AirPassengers

Kevin-Chen0 commented Jan 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ourownstory Jan 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kevin-Chen0 Jan 13, 2023 • edited Loading

Choose a reason for hiding this comment

ourownstory Jan 13, 2023 • edited Loading

Choose a reason for hiding this comment

ourownstory left a comment

Choose a reason for hiding this comment

Kevin-Chen0 commented Dec 17, 2022 •

edited

Loading

codecov-commenter commented Dec 17, 2022 •

edited

Loading

github-actions bot commented Dec 17, 2022 •

edited

Loading

ourownstory Jan 13, 2023 •

edited

Loading

Kevin-Chen0 Jan 13, 2023 •

edited

Loading

ourownstory Jan 13, 2023 •

edited

Loading