Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] Added four blocksparsity layouts #320

Commits on Jun 1, 2022

  1. added four blocksparsity layouts

    Four sparsity layouts from DeepSpeed are now available for blocksparse attention on xFormer:
    Fixed
    BSLongformer
    BigBird
    Variable
    sparsity_configs.py (https://fburl.com/code/s2n7x8gs) contains flexible objects with many parameters. The default parameters can be invoked through the quick_ functionality in attention_patterns.py (https://fburl.com/code/hya0t9e7) the produced layouts can be turned into a pattern with layout_to_pattern function (https://fburl.com/code/1qmsntyj)
    Detailed notes for the task: https://docs.google.com/document/d/1cBlZeccvphI-d5avLkgKwZ4ScXQM1q6igqvlyFDXdDc/edit?usp=sharing
    igormolybogFB committed Jun 1, 2022
    Configuration menu
    Copy the full SHA
    26fbdd9 View commit details
    Browse the repository at this point in the history
  2. pre-commit linting fixed

    linting errors fixed for std-blocksparsity-attention-layouts
    igormolybogFB committed Jun 1, 2022
    Configuration menu
    Copy the full SHA
    9a5f696 View commit details
    Browse the repository at this point in the history
  3. Added copyright, fixed block

    Added copyright
    fixed block -> block_size
    removed DeepSpeed TODO
    igormolybogFB committed Jun 1, 2022
    Configuration menu
    Copy the full SHA
    7a9c9a3 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2022

  1. Configuration menu
    Copy the full SHA
    1c54759 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b942065 View commit details
    Browse the repository at this point in the history
  3. Fixed a unittest

    igormolybogFB committed Jun 2, 2022
    Configuration menu
    Copy the full SHA
    c5e7415 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2aed52c View commit details
    Browse the repository at this point in the history
  5. removed .DS_Store

    igormolybogFB committed Jun 2, 2022
    Configuration menu
    Copy the full SHA
    878ea72 View commit details
    Browse the repository at this point in the history