[minor] Dependency management with poetry

[noxan]

🔬 Background

• Poetry is a modern python dependency manager unifying all requirements and distribution related configuration

🔮 Key changes

• Review all of our dependencies and remove unused ones (Review and remove unused dependencies #1201)
• Lock all package version numbers to their latest feature release which is compatible with our minimum python version of 3.7.1 (Project dependencies may have API risk issues #911)

❓ Discussion points

1. Any reasons not to remove the requirements/*.txt setup?

📓 Open tasks

1. Migrate the github actions to the new poetry setup (away from pip)

✨ Why poetry?

1. It ensures package versions are compatible with the python version
2. Ensures the right package dependency version and their dependencies are installed with its lock-file
3. Automatically manages the virtual environment setup
4. Integrates the publish workflow (pypi)
5. Supports different dependency groups (e.g. tests, docs) and unifies them all into a single configuration file
6. Easily keep track of all outdated packages (with poetry list --outdated) and update them accordingly

[github-actions]

Model Benchmark

Benchmark	Metric	main	current	diff
AirPassengers	MAE_val	15.2698	15.2698	0.0%	✅
AirPassengers	RMSE_val	19.4209	19.4209	0.0%	✅
AirPassengers	Loss_val	0.00195	0.00195	0.0%	✅
AirPassengers	MAE	9.82902	9.82902	0.0%	✅
AirPassengers	RMSE	11.7005	11.7005	0.0%	✅
AirPassengers	Loss	0.00056	0.00056	0.0%	✅
AirPassengers	time	4.26428	4.51	5.76%	⚠️
YosemiteTemps	MAE_val	1.71298	1.71298	0.0%	✅
YosemiteTemps	RMSE_val	2.2607	2.2607	0.0%	✅
YosemiteTemps	Loss_val	0.00095	0.00095	0.0%	✅
YosemiteTemps	MAE	1.45187	1.45187	0.0%	✅
YosemiteTemps	RMSE	2.16721	2.16721	0.0%	✅
YosemiteTemps	Loss	0.00066	0.00066	0.0%	✅
YosemiteTemps	time	100.316	104.55	4.22%	⚠️
PeytonManning	MAE_val	0.58159	0.58159	0.0%	✅
PeytonManning	RMSE_val	0.72216	0.72216	0.0%	✅
PeytonManning	Loss_val	0.01239	0.01239	0.0%	✅
PeytonManning	MAE	0.41671	0.41671	0.0%	✅
PeytonManning	RMSE	0.55961	0.55961	0.0%	✅
PeytonManning	Loss	0.00612	0.00612	0.0%	✅
PeytonManning	time	11.8229	12.73	7.67%	❌

Model training plots

Model Training

PeytonManning

YosemiteTemps

AirPassengers

[ourownstory]

@noxan I concur that this would be the right thing to do if our package would not be used within other packages or in environments with other packages.
However, tightening the requirements more than critically necessary will cause issues for libraries such as Darts using our package, as you can see from #1129
Also, users who want to use multiple different models in the same environment may run into issues.

Does poetry allow us to relax, rather than tighten the dependency bounds?

A small point is the familiarity most developers/users have with pip vs poetry.

[noxan]

@ourownstory

However, tightening the requirements more than critically necessary will cause issues for libraries such as Darts using our package, as you can see from #1129

Also I agree with your argument of not defining them to tightly as it creates issues. But having unbound dependency versions creates other kinds of issues, e.g. our package breaks if another dependency introduces a breaking change. And in my experience it is better to be explicit about versions, it is easy to report a too tightly bound dependency and relax it. In contrary you can spend hours on figuring out which versions are incompatible, and why which package breaks. Especially if they are not explicitly defined, everything works fine, you recreate your development environment or a new developer joins the team and nothing works anymore - because the dependency versions luckily overlapped during your first project setup, just now things changed and nobody has a clue what...

Does poetry allow us to relax, rather than tighten the dependency bounds?

How tightly we define our dependencies is independent of poetry or pip. They both allow the same options.

Also, users who want to use multiple different models in the same environment may run into issues.

Into what kind of issues? That package versions are not compatible? At least it would be explicit then, good look trying to figure out which exact version of a dependency (3rd party) of your dependencies are incompatible to another.

A small point is the familiarity most developers/users have with pip vs poetry.

Agree - yet poetry also makes a lot of things a lot easier

[noxan]

Another thing which I became aware of: we use a lot of dependencies which are actually not supporting our python version anymore, e.g. pandas dropped python 3.7 support with version 1.4. Our requirements allows anything pandas>=1.0.4 - apparently it still works but how knows for how long they break with python 3.7 - it is not even officially supported anymore.

Another example: pytorch is not compatible with python 3.11 yet - so we are neither.

[LeonieFreisinger]

LGTM @noxan Please have a look at the failing test again

[codecov-commenter]

Codecov Report

Merging #1202 (fa98462) into main (70ecb9e) will increase coverage by 0.10%.
The diff coverage is n/a.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##             main    #1202      +/-   ##
==========================================
+ Coverage   89.67%   89.77%   +0.10%     
==========================================
  Files          38       38              
  Lines        4977     5055      +78     
==========================================
+ Hits         4463     4538      +75     
- Misses        514      517       +3     

see 4 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.