Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uncertainty: Conformal Prediction V1.1 - add Conformal class to conformal_prediction.py and rename file to conformal.py #1074

Merged
merged 15 commits into from
Jan 4, 2023

Conversation

Kevin-Chen0
Copy link
Collaborator

@Kevin-Chen0 Kevin-Chen0 commented Dec 18, 2022

🔬 Background

🔮 Key changes

  • Add Conformal class to conformal_prediction.py, rename the file to conformal.py, and refactor the file.

📋 Review Checklist

  • I have performed a self-review of my own code.
  • I have commented my code, added docstrings and data types to function definitions.
  • I have added pytests to check whether my feature / fix works.

Please make sure to follow our best practices in the Contributing guidelines.

@Kevin-Chen0 Kevin-Chen0 added uncertainty status: needs update PR has outstanding comment(s) or PR test(s) that need to be resolved priority:P2 Medium priority labels Dec 18, 2022
@Kevin-Chen0 Kevin-Chen0 added this to the Release 0.5.1 milestone Dec 18, 2022
@Kevin-Chen0 Kevin-Chen0 self-assigned this Dec 18, 2022
@github-actions
Copy link

github-actions bot commented Dec 18, 2022

3e7b56a

Model Benchmark

Benchmark Metric main current diff
AirPassengers MAE_val 15.2698 15.2698 0.0%
AirPassengers RMSE_val 19.4209 19.4209 0.0%
AirPassengers Loss_val 0.00195 0.00195 0.0%
AirPassengers MAE 9.82902 9.82902 0.0%
AirPassengers RMSE 11.7005 11.7005 0.0%
AirPassengers Loss 0.00056 0.00056 0.0%
AirPassengers time 6.14397 5.15 -16.18% 🎉
YosemiteTemps MAE_val 1.72948 1.72949 0.0%
YosemiteTemps RMSE_val 2.27386 2.27386 0.0%
YosemiteTemps Loss_val 0.00096 0.00096 0.0%
YosemiteTemps MAE 1.45189 1.45189 0.0%
YosemiteTemps RMSE 2.16631 2.16631 0.0%
YosemiteTemps Loss 0.00066 0.00066 0.0%
YosemiteTemps time 142.999 115.22 -19.43% 🎉
PeytonManning MAE_val 0.64636 0.64636 0.0%
PeytonManning RMSE_val 0.79276 0.79276 0.0%
PeytonManning Loss_val 0.01494 0.01494 0.0%
PeytonManning MAE 0.42701 0.42701 0.0%
PeytonManning RMSE 0.57032 0.57032 0.0%
PeytonManning Loss 0.00635 0.00635 0.0%
PeytonManning time 16.8242 13.76 -18.21% 🎉
Model training plots

Model Training

PeytonManning

YosemiteTemps

AirPassengers

@codecov-commenter
Copy link

codecov-commenter commented Dec 19, 2022

Codecov Report

Merging #1074 (f54ec10) into main (9272959) will increase coverage by 0.01%.
The diff coverage is 94.64%.

@@            Coverage Diff             @@
##             main    #1074      +/-   ##
==========================================
+ Coverage   90.12%   90.13%   +0.01%     
==========================================
  Files          21       21              
  Lines        4788     4796       +8     
==========================================
+ Hits         4315     4323       +8     
  Misses        473      473              
Impacted Files Coverage Δ
neuralprophet/conformal.py 94.00% <94.00%> (ø)
neuralprophet/forecaster.py 87.86% <100.00%> (-0.02%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@Kevin-Chen0 Kevin-Chen0 added status: needs review PR needs to be reviewed by Reviewer(s) and removed status: needs update PR has outstanding comment(s) or PR test(s) that need to be resolved labels Dec 23, 2022
Copy link
Collaborator

@noxan noxan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, a few minor comments


alpha: float
method: str
quantiles: list = None
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
quantiles: list = None
quantiles: Optional[list] = None

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can this be None?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If None, NeuralProphet will automatically set it to [] behind-the-scene. Then, if 0.5 doesn't exist in the input quantiles list, then NP will automatically add it to the front of the list. Therefore, default None will become [0.5].

neuralprophet/conformal.py Outdated Show resolved Hide resolved
Comment on lines 1 to 3
# import logging
# from typing import Callable, List, Optional, Type, Union

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# import logging
# from typing import Callable, List, Optional, Type, Union

Remove unused imports

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# import logging
# from typing import Callable, List, Optional, Type, Union
from typing import Optional

Or import Optional for later usage

neuralprophet/conformal.py Outdated Show resolved Hide resolved
Kevin-Chen0 and others added 3 commits January 3, 2023 18:44
Co-authored-by: Richard Stromer <noxan@users.noreply.github.com>
Co-authored-by: Richard Stromer <noxan@users.noreply.github.com>
@Kevin-Chen0 Kevin-Chen0 added status: ready PR is ready to be merged and removed status: needs review PR needs to be reviewed by Reviewer(s) labels Jan 4, 2023
@Kevin-Chen0 Kevin-Chen0 merged commit b91c3fe into main Jan 4, 2023
@Kevin-Chen0 Kevin-Chen0 deleted the refactor/conformal-class branch January 4, 2023 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority:P2 Medium priority status: ready PR is ready to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants