[fix] Layer positions labelling & layernorm #348
Conversation
facebook-github-bot
added
the
CLA Signed
| @@ -149,10 +149,12 @@ def __init__( | |||
| for i in range(config.num_layers): | |||
| # Label where this layer is in the stack | |||
| # (for instance useful for the positional encoding, or late layer norm) | |||
| if i > 0: | |||
There was a problem hiding this comment.
this would count within the repeated layers, not the overall layer stack
blefaudeux
force-pushed
the
hierachical_models_improvement
branch
from
e5f22e4
to
ebc4f6f
Compare
blefaudeux
force-pushed
the
fix_layer_positions
branch
from
67ba72f
to
f8f4972
Compare
| @@ -175,10 +175,7 @@ def __init__(self, config: xFormerEncoderConfig, **kwargs): | |||
| # Optional patch embedding | |||
| self.patch_emb: Optional[nn.Module] = None | |||
|
|
|||
| if ( | |||
There was a problem hiding this comment.
this was not respecting the config, while trying to do the right thing: if the config was asking for a patch embedding, but the layer was not first, it would not be instantiated. In retrospect I think that it's risky, not doing what the API says it will do, plus it only worked in practice because is_first() was often wrong. I think now that it's better to respect the config no matter what, and not silently diverge
Codecov Report
@@ Coverage Diff @@
## hierachical_models_improvement #348 +/- ##
===============================================================
Coverage 93.95% 93.95%
===============================================================
Files 70 70
Lines 3984 3984
===============================================================
Hits 3743 3743
Misses 241 241
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
) * handling different normalizations + layer repetition * bugfix localizing the layers in the stack (#348) * renaming the layer_norm_style param when building from config Co-authored-by: Benjamin Lefaudeux <lefaudeux@Benjamins-MacBook-Pro.local>
What does this PR do?
Fixes #347
Before submitting
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.