Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure that serialized data is measured correctly #7593
Ensure that serialized data is measured correctly #7593
Changes from 7 commits
d436ee0
9584b28
4fe360d
690ec48
d3bc7ec
3d0fc9c
dafc08a
1fbadf8
6c2f4ad
52eee9e
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test logic is now slightly different but I believe more robust. We don't truly care about the offloading part but rather that there is an await during deserialization. Therefore, I'll only patch the offload method. To ensure that we're truly in the task spec deserialization I put in the
istask
guard aboveThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
somehow, this actually shutdown the actual offload threadpool, not just the mock, i.e. nothing in our test suite was using the offloader threadpool after this test ran 🤯
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not entirely understand why this is shutting down the actual threadpool but I don't care. I removed the mock and it works now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just an attempt to fix the error I'm seeing
Two reasons why I believe this should be removed regardless of whether this is a fix or not
finalize
docs I'm not even sure if this callback is ever triggeredsince
_offload_executor
is a module global and unless it has been replaced by None, it has no chance of being GCed/finalizedThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is not responsible after all but I still suggest to remove this line
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's interesting, later in the docs it states:
Notably the last part, which I suppose is part of the unneeded bit of this line.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, that note is interesting as well, i.e. this finalize is useless for many reasons
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I opened #7639
Our code base is riddled with this pattern
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one line makes me very nervous. Everything you wrote makes perfect sense, but just in case you're wrong, could you move it to its own PR so that it's not ending up in the release? It has the potential of leaving workers stuck on shutdown, and it also has the potential of different behaviour on different Python versions and on different OSs, so I believe some thorough testing is in order.