Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[refactor] Uncertainty streamline naming #1262

Merged
merged 23 commits into from
Apr 29, 2023

Conversation

leoniewgnr
Copy link
Collaborator

@leoniewgnr leoniewgnr commented Apr 13, 2023

🔬 Background

The forecast data frames gets a new column for every new uncertainty prediction method which leads to a very large data frame.
Fixes issue #1261

🔮 Key changes

The initial uncertainty intervals are overwritten with the new ones when a new uncertainty prediction method is used.

📋 Review Checklist

  • I have performed a self-review of my own code.
  • I have commented my code, added docstrings and data types to function definitions.
  • I have added pytests to check whether my feature / fix works.

@leoniewgnr leoniewgnr self-assigned this Apr 13, 2023
@leoniewgnr leoniewgnr added this to the Release 0.6.0 milestone Apr 13, 2023
@leoniewgnr leoniewgnr linked an issue Apr 13, 2023 that may be closed by this pull request
@leoniewgnr leoniewgnr added the status: in development Pull requests which are in development label Apr 13, 2023
@github-actions
Copy link

github-actions bot commented Apr 13, 2023

Model Benchmark

Benchmark Metric main current diff
AirPassengers MAE_val 13.0626 13.0626 0.0%
AirPassengers RMSE_val 15.9453 15.9453 0.0%
AirPassengers Loss_val 0.00131 0.00131 0.0%
AirPassengers MAE 9.88156 9.88156 0.0%
AirPassengers RMSE 11.7354 11.7354 0.0%
AirPassengers Loss 0.00052 0.00052 0.0%
AirPassengers time 5.18982 4.92 -5.2% 🎉
PeytonManning MAE_val 0.58159 0.58159 0.0%
PeytonManning RMSE_val 0.72216 0.72216 0.0%
PeytonManning Loss_val 0.01239 0.01239 0.0%
PeytonManning MAE 0.41671 0.41671 0.0%
PeytonManning RMSE 0.55961 0.55961 0.0%
PeytonManning Loss 0.00612 0.00612 0.0%
PeytonManning time 12.7264 11.96 -6.02% 🎉
YosemiteTemps MAE_val 1.3442 1.3442 0.0%
YosemiteTemps RMSE_val 2.00245 2.00245 0.0%
YosemiteTemps Loss_val 0.00077 0.00077 0.0%
YosemiteTemps MAE 1.3192 1.3192 0.0%
YosemiteTemps RMSE 2.13518 2.13518 0.0%
YosemiteTemps Loss 0.00064 0.00064 0.0%
YosemiteTemps time 67.5437 61.65 -8.73% 🎉
Model training plots

Model Training

PeytonManning

YosemiteTemps

AirPassengers

@codecov-commenter
Copy link

codecov-commenter commented Apr 13, 2023

Codecov Report

Merging #1262 (07d1668) into main (209129e) will decrease coverage by 0.05%.
The diff coverage is 96.51%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##             main    #1262      +/-   ##
==========================================
- Coverage   89.84%   89.79%   -0.05%     
==========================================
  Files          38       38              
  Lines        5149     5126      -23     
==========================================
- Hits         4626     4603      -23     
  Misses        523      523              
Impacted Files Coverage Δ
neuralprophet/forecaster.py 87.38% <86.66%> (-0.04%) ⬇️
neuralprophet/plot_forecast_plotly.py 87.36% <92.30%> (+0.05%) ⬆️
neuralprophet/plot_forecast_matplotlib.py 84.47% <100.00%> (-0.81%) ⬇️
neuralprophet/uncertainty.py 98.69% <100.00%> (+1.36%) ⬆️

@leoniewgnr leoniewgnr changed the title Uncertainty streamline naming [refactor] Uncertainty streamline naming Apr 13, 2023
@leoniewgnr leoniewgnr marked this pull request as ready for review April 14, 2023 00:02
@leoniewgnr leoniewgnr added status: needs review PR needs to be reviewed by Reviewer(s) and removed status: in development Pull requests which are in development labels Apr 14, 2023
@ourownstory ourownstory modified the milestones: Release 0.6.0, 0.6.1 Apr 14, 2023
neuralprophet/plot_forecast_matplotlib.py Show resolved Hide resolved
neuralprophet/uncertainty.py Show resolved Hide resolved
neuralprophet/uncertainty.py Outdated Show resolved Hide resolved
@Kevin-Chen0 Kevin-Chen0 added status: needs update PR has outstanding comment(s) or PR test(s) that need to be resolved and removed status: needs review PR needs to be reviewed by Reviewer(s) labels Apr 17, 2023
neuralprophet/uncertainty.py Show resolved Hide resolved
neuralprophet/forecaster.py Show resolved Hide resolved
@leoniewgnr
Copy link
Collaborator Author

Basic user:

method = "naive"  # or "cqr" for a more sophisticated method, see uncertainty tutorial
conformal_forecast = m.conformal_predict(train_df, cal_df, alpha=0.1, method=method)
fig = m.highlight_nth_step_ahead_of_each_forecast(1).plot(conformal_forecast)

--> returns only y, yhat and the updated uncertainty intervals
--> plot shows only the forecast and one uncertainty interval around it

Advanced user:

method = "naive"  # or "cqr" for a more sophisticated method, see uncertainty tutorial
conformal_forecast = m.conformal_predict(train_df, cal_df, alpha=0.1, method=method, show_qr=True)
fig = m.conformal_plot(conformal_forecast, n_highlight =1)

--> returns updated uncertainty intervals (conformal prediction: naive or cqr) and previous quantile regression intervals
--> plot shows previous qr intervals in light blue and additional lines for conformal prediction

@leoniewgnr leoniewgnr added status: needs review PR needs to be reviewed by Reviewer(s) and removed status: needs update PR has outstanding comment(s) or PR test(s) that need to be resolved labels Apr 21, 2023
neuralprophet/__init__.py Outdated Show resolved Hide resolved
neuralprophet/forecaster.py Outdated Show resolved Hide resolved
neuralprophet/forecaster.py Outdated Show resolved Hide resolved
neuralprophet/forecaster.py Outdated Show resolved Hide resolved
@Kevin-Chen0 Kevin-Chen0 added the status: needs update PR has outstanding comment(s) or PR test(s) that need to be resolved label Apr 21, 2023
@leoniewgnr leoniewgnr removed the status: needs review PR needs to be reviewed by Reviewer(s) label Apr 21, 2023
@leoniewgnr leoniewgnr added status: needs review PR needs to be reviewed by Reviewer(s) and removed status: needs update PR has outstanding comment(s) or PR test(s) that need to be resolved labels Apr 21, 2023
Copy link
Collaborator

@LeonieFreisinger LeonieFreisinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@LeonieFreisinger LeonieFreisinger merged commit 9e868ea into main Apr 29, 2023
@LeonieFreisinger LeonieFreisinger deleted the uncertainty-streamline-naming branch April 29, 2023 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: needs review PR needs to be reviewed by Reviewer(s)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Refactor] Streamline uncertainty naming of quantile
6 participants