Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check all param groups for flat parameters in FSDP #17914

Merged

Conversation

schmidt-ai
Copy link
Contributor

Closes #17817

@github-actions github-actions bot added the fabric lightning.fabric.Fabric label Jun 24, 2023
@awaelchli awaelchli changed the title Patch/ optimizer has flat params Check all param groups for flat parameters in FSDP Jun 26, 2023
@awaelchli awaelchli added strategy: fsdp Fully Sharded Data Parallel bug Something isn't working labels Jun 26, 2023
@awaelchli awaelchli added this to the 2.0.x milestone Jun 26, 2023
@mergify mergify bot added the ready PRs ready to be merged label Jun 26, 2023
@carmocca carmocca enabled auto-merge (squash) June 26, 2023 02:09
@carmocca carmocca merged commit 9337d1c into Lightning-AI:master Jun 26, 2023
@schmidt-ai schmidt-ai deleted the patch/_optimizer_has_flat_params branch July 5, 2023 16:33
Borda pushed a commit that referenced this pull request Jul 7, 2023
Co-authored-by: awaelchli <aedu.waelchli@gmail.com>

(cherry picked from commit 9337d1c)
lantiga pushed a commit that referenced this pull request Jul 10, 2023
Co-authored-by: awaelchli <aedu.waelchli@gmail.com>

(cherry picked from commit 9337d1c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working fabric lightning.fabric.Fabric ready PRs ready to be merged strategy: fsdp Fully Sharded Data Parallel
Projects
None yet
Development

Successfully merging this pull request may close these issues.

FSDP: _optimizer_has_flat_params only checks first parameter group
3 participants