-
Notifications
You must be signed in to change notification settings - Fork 221
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SWDEV-300279][MSRCHA-140] Batchnorm backward convergence fix #1116
Conversation
This comment has been minimized.
This comment has been minimized.
@@ -865,7 +865,7 @@ void BatchNormBackward(Handle& handle, | |||
// N*H*W < 32M and H*W > 1024, use batchnorm variant#1 implementation which parallelize | |||
// work groups over channels and loop through NHW. | |||
//************************************************************************************************* | |||
if((in_nhw < (32 * 1024 * 1024) && in_cstride > 1024) || (n > 768)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why batch sizes greater 768 will have problem parallelize work groups over channels and loop through NHW? Just curious how this has fixed the converge issue. Thanks1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should always check HW (in_cstride) which is missing in this case. We were using batchnorm variant#1 for batch sizes > 768 which is having issues for these batch sizes when HW is small. I have corrected it to use correct variant. This bug might have been introduced inadvertently. I fixed it.
Ping @ce1adon for review. One line change only. Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CI has passed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to add a regression test.
This fix will address batchnorm backward convergence issue for batch sizes greater 768.