Add testing for PyTorch 2.4 (Fabric) #20028

awaelchli · 2024-06-30T13:31:38Z

What does this PR do?

Adds PyTorch 2.4 RC to the testing matrix of Fabric (PyTorch Lightning in follow up).

Notes:

Decided to bump the minimum required version of ModelParallelStrategy to PyTorch >= 2.4. The APIs are anyway still experimental. To simplify maintenance on our end, let's only support the latest version of PyTorch for now until their parallelism/dtensor support is stable.
The FSDP state_dict APIs are emitting FutureWarnings/deprecation warnings. It will be a greater effort to switch to the new ones (the ModelParallelStrategy already uses the new ones) and provide backward compatibility for older PyTorch versions. So we're not doing it in this PR.
Not yet adding the test entries to the checkgroup. If something fails as PyTorch 2.4 rc updates, we don't block development. We will add them as required tests once PyTorch 2.4 stable is out.

cc @carmocca @Borda @justusschock @awaelchli

codecov · 2024-06-30T13:41:21Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59%. Comparing base (14493c0) to head (300fc6f).
Report is 74 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (14493c0) and HEAD (300fc6f). Click for more details.

HEAD has 18 uploads less than BASE

Flag BASE (14493c0) HEAD (300fc6f)

examples 6 0

app 6 0

lightning_app 6 0

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #20028     +/-   ##
=========================================
- Coverage      71%      59%    -12%     
=========================================
  Files         427      422      -5     
  Lines       35460    35362     -98     
=========================================
- Hits        25234    20754   -4480     
- Misses      10226    14608   +4382

for more information, see https://pre-commit.ci

…ghtning into tests/pytorch-2.4-fabric

github-actions · 2024-06-30T14:16:08Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-13, lightning, 3.8, 2.1, oldest)	success	✅
pl-cpu (macOS-14, lightning, 3.10, 2.1)	success	✅
pl-cpu (macOS-14, lightning, 3.10, 2.2)	success	✅
pl-cpu (macOS-14, lightning, 3.10, 2.3)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 2.1, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.2)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.3)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 2.1, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.1)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.2)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.3)	success	✅
pl-cpu (macOS-14, pytorch, 3.8, 2.1)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 2.1)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 2.1)	success	✅
pl-cpu (macOS-12, pytorch, 3.11, 2.1)	success	✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.1)	success	✅
pl-cpu (windows-2022, pytorch, 3.11, 2.1)	success	✅

These checks are required after the changes to requirements/fabric/base.txt, requirements/fabric/examples.txt, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/model_parallel.py, src/lightning/fabric/utilities/imports.py, src/lightning/fabric/utilities/testing/_runif.py, tests/tests_pytorch/strategies/test_model_parallel.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs) (testing Lightning \| latest)	success	✅
pytorch-lightning (GPUs) (testing PyTorch \| latest)	success	✅

These checks are required after the changes to tests/tests_pytorch/strategies/test_model_parallel.py, requirements/fabric/base.txt, requirements/fabric/examples.txt, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/model_parallel.py, src/lightning/fabric/utilities/imports.py, src/lightning/fabric/utilities/testing/_runif.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to requirements/fabric/base.txt, requirements/fabric/examples.txt, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/model_parallel.py, src/lightning/fabric/utilities/imports.py, src/lightning/fabric/utilities/testing/_runif.py.

🟢 fabric: Docs

Check ID	Status
docs-make (fabric, doctest)	success	✅
docs-make (fabric, html)	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/model_parallel.py, src/lightning/fabric/utilities/imports.py, src/lightning/fabric/utilities/testing/_runif.py, requirements/fabric/base.txt, requirements/fabric/examples.txt.

🟢 pytorch_lightning: Docker

Check ID	Status
build-cuda (3.10, 2.1, 12.1.0)	success	✅
build-cuda (3.10, 2.2, 12.1.0)	success	✅
build-cuda (3.11, 2.1, 12.1.0)	success	✅
build-cuda (3.11, 2.2, 12.1.0)	success	✅
build-pl (3.10, 2.1, 12.1.0)	success	✅
build-pl (3.10, 2.2, 12.1.0)	success	✅
build-pl (3.11, 2.1, 12.1.0)	success	✅
build-pl (3.11, 2.2, 12.1.0)	success	✅

These checks are required after the changes to requirements/fabric/base.txt, requirements/fabric/examples.txt.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-13, lightning, 3.8, 2.1, oldest)	success	✅
fabric-cpu (macOS-14, lightning, 3.11, 2.1)	success	✅
fabric-cpu (macOS-14, lightning, 3.11, 2.2)	success	✅
fabric-cpu (macOS-14, lightning, 3.10, 2.3)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 2.1, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.1)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.2)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.3)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 2.1, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.1)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.2)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.3)	success	✅
fabric-cpu (macOS-14, fabric, 3.8, 2.1)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 2.1)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 2.1)	success	✅
fabric-cpu (macOS-12, fabric, 3.11, 2.1)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.1)	success	✅
fabric-cpu (windows-2022, fabric, 3.11, 2.1)	success	✅

These checks are required after the changes to requirements/fabric/base.txt, requirements/fabric/examples.txt, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/model_parallel.py, src/lightning/fabric/utilities/imports.py, src/lightning/fabric/utilities/testing/_runif.py, tests/tests_fabric/conftest.py, tests/tests_fabric/plugins/precision/test_amp.py, tests/tests_fabric/plugins/precision/test_amp_integration.py, tests/tests_fabric/plugins/precision/test_bitsandbytes.py, tests/tests_fabric/strategies/test_dp.py, tests/tests_fabric/strategies/test_fsdp_integration.py, tests/tests_fabric/strategies/test_model_parallel.py, tests/tests_fabric/strategies/test_model_parallel_integration.py, tests/tests_fabric/test_connector.py, tests/tests_fabric/test_wrappers.py, .github/workflows/ci-tests-fabric.yml.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs) (testing Fabric \| latest)	success	✅
lightning-fabric (GPUs) (testing Lightning \| latest)	success	✅

These checks are required after the changes to .azure/gpu-tests-fabric.yml, requirements/fabric/base.txt, requirements/fabric/examples.txt, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/model_parallel.py, src/lightning/fabric/utilities/imports.py, src/lightning/fabric/utilities/testing/_runif.py, tests/tests_fabric/conftest.py, tests/tests_fabric/plugins/precision/test_amp.py, tests/tests_fabric/plugins/precision/test_amp_integration.py, tests/tests_fabric/plugins/precision/test_bitsandbytes.py, tests/tests_fabric/strategies/test_dp.py, tests/tests_fabric/strategies/test_fsdp_integration.py, tests/tests_fabric/strategies/test_model_parallel.py, tests/tests_fabric/strategies/test_model_parallel_integration.py, tests/tests_fabric/test_connector.py, tests/tests_fabric/test_wrappers.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to requirements/fabric/base.txt, requirements/fabric/examples.txt, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/model_parallel.py, src/lightning/fabric/utilities/imports.py, src/lightning/fabric/utilities/testing/_runif.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.11)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.11)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.11)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.11)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.11)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.11)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.11)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.11)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.11)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.11)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.11)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.11)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.11)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.11)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.11)	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/model_parallel.py, src/lightning/fabric/utilities/imports.py, src/lightning/fabric/utilities/testing/_runif.py, requirements/fabric/base.txt, requirements/fabric/examples.txt.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

.azure/gpu-tests-fabric.yml

Add testing for PyTorch 2.4 (Fabric)

4ff0ea8

github-actions bot added ci Continuous Integration fabric lightning.fabric.Fabric dependencies Pull requests that update a dependency file dockers labels Jun 30, 2024

awaelchli added this to the 2.4 milestone Jun 30, 2024

typing

9e62bf6

awaelchli and others added 10 commits June 30, 2024 15:42

debug

09d4aaf

skip

a2da137

typing

d55a478

deprecated state_dict

ebdc431

[pre-commit.ci] auto fixes from pre-commit.com hooks

6ea0872

for more information, see https://pre-commit.ci

update

c018e4c

Merge branch 'tests/pytorch-2.4-fabric' of github.com:Lightning-AI/li…

001aa26

…ghtning into tests/pytorch-2.4-fabric

docs

c147d10

filterwarnings

fe8d143

futurewarnings

9909adb

awaelchli marked this pull request as ready for review June 30, 2024 14:15

awaelchli requested review from lantiga, Borda, tchaton, justusschock and ethanwharris as code owners June 30, 2024 14:15

reset docker build

9243486

Borda reviewed Jun 30, 2024

View reviewed changes

.azure/gpu-tests-fabric.yml Outdated Show resolved Hide resolved

delete obsolete test

2f8402f

github-actions bot added the pl Generic label for PyTorch Lightning package label Jun 30, 2024

store python version in env variable

116fec6

awaelchli force-pushed the tests/pytorch-2.4-fabric branch from 4f92d45 to 116fec6 Compare June 30, 2024 17:22

awaelchli added 2 commits June 30, 2024 22:52

exclude windows

a83b75b

Merge branch 'master' into tests/pytorch-2.4-fabric

2c75547

mergify bot added the has conflicts label Jun 30, 2024

Merge branch 'master' into tests/pytorch-2.4-fabric

8a5069e

mergify bot removed the has conflicts label Jun 30, 2024

fix bad merge resolution

300fc6f

awaelchli requested a review from Borda July 1, 2024 11:54

justusschock approved these changes Jul 2, 2024

View reviewed changes

awaelchli merged commit 693c21a into master Jul 2, 2024
118 checks passed

awaelchli deleted the tests/pytorch-2.4-fabric branch July 2, 2024 22:01

awaelchli mentioned this pull request Jul 13, 2024

Update PyTorch 2.4 tests #20079

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add testing for PyTorch 2.4 (Fabric) #20028

Add testing for PyTorch 2.4 (Fabric) #20028

awaelchli commented Jun 30, 2024 •

edited

Loading

codecov bot commented Jun 30, 2024 •

edited

Loading

github-actions bot commented Jun 30, 2024 •

edited

Loading

Add testing for PyTorch 2.4 (Fabric) #20028

Add testing for PyTorch 2.4 (Fabric) #20028

Conversation

awaelchli commented Jun 30, 2024 • edited Loading

What does this PR do?

codecov bot commented Jun 30, 2024 • edited Loading

Codecov Report

github-actions bot commented Jun 30, 2024 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

awaelchli commented Jun 30, 2024 •

edited

Loading

codecov bot commented Jun 30, 2024 •

edited

Loading

github-actions bot commented Jun 30, 2024 •

edited

Loading