Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#24789][prism] internal/worker + tentative data #25478

Merged
merged 3 commits into from
Feb 20, 2023

Conversation

lostluck
Copy link
Contributor

Add the GRPC FnAPI implementation that sits between SDK harnesses and the runner on workers.

For Prism's initial implementation.

See #24789.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI.

@lostluck
Copy link
Contributor Author

R: @jrmccluskey

@github-actions
Copy link
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@codecov
Copy link

codecov bot commented Feb 18, 2023

Codecov Report

Merging #25478 (aae628b) into master (8283e4f) will decrease coverage by 0.10%.
The diff coverage is 66.00%.

@@            Coverage Diff             @@
##           master   #25478      +/-   ##
==========================================
- Coverage   72.93%   72.84%   -0.10%     
==========================================
  Files         748      755       +7     
  Lines       99366   100421    +1055     
==========================================
+ Hits        72476    73151     +675     
- Misses      25523    25852     +329     
- Partials     1367     1418      +51     
Flag Coverage Δ
go 52.25% <66.00%> (+0.40%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../go/pkg/beam/runners/prism/internal/engine/data.go 0.00% <0.00%> (ø)
...o/pkg/beam/runners/prism/internal/worker/worker.go 63.42% <63.42%> (ø)
...o/pkg/beam/runners/prism/internal/worker/bundle.go 90.24% <90.24%> (ø)
sdks/go/pkg/beam/core/runtime/exec/sdf.go 66.12% <0.00%> (-4.89%) ⬇️
sdks/go/pkg/beam/io/textio/textio.go 54.21% <0.00%> (-0.95%) ⬇️
sdks/go/pkg/beam/core/runtime/xlangx/expand.go 0.00% <0.00%> (ø)
sdks/go/pkg/beam/runners/prism/internal/coders.go 91.39% <0.00%> (ø)
...pkg/beam/runners/prism/internal/engine/strategy.go 50.00% <0.00%> (ø)
...o/pkg/beam/core/runtime/exec/sdf_invokers_arity.go 31.91% <0.00%> (ø)
...beam/runners/prism/internal/jobservices/metrics.go 70.83% <0.00%> (ø)
... and 7 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@lostluck
Copy link
Contributor Author

R: @johannaojeling

Would you mind helping me usher in the Prism runner before the 2.46 cut?

I'd love for it to be in a usable state in that release, and to get it so others can possible help complete it sooner.

In particular for reviews at this stage, it would be great to get doc/ code ordering changes so it's not entirely confusing to readers. New functionality/bug fixes will become TODOs though, rather than completing them, since they may have strong follow on effects to the branch.

@github-actions
Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@johannaojeling
Copy link
Contributor

Of course, I'll have a look at this tomorrow! (UTC+1)

Copy link
Contributor

@johannaojeling johannaojeling left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! At this point I don't have enough knowledge to provide a meaningful review with regards to functionality (but I'm interested to learn and like to read your code, so thanks for tagging me). I left a few comments for some minor things you may or may not want to address 😊 well done!

sdks/go/pkg/beam/runners/prism/internal/worker/worker.go Outdated Show resolved Hide resolved
func toSlogSev(sev fnpb.LogEntry_Severity_Enum) slog.Level {
switch sev {
case fnpb.LogEntry_Severity_TRACE:
return slog.Level(-8) //
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return slog.Level(-8) //
return slog.Level(-8)

sdks/go/pkg/beam/runners/prism/internal/worker/worker.go Outdated Show resolved Hide resolved
sdks/go/pkg/beam/runners/prism/internal/worker/worker.go Outdated Show resolved Hide resolved
@lostluck
Copy link
Contributor Author

These were excellent! Thank you very much.

When there's nearly 8000 lines to get in (including documentation, tests and test functions), it really adds up, so splitting things up is hopefully going to make it easier to review.

Unfortunately, we're now getting into the complicated stuff.

@lostluck lostluck merged commit ca1ec25 into apache:master Feb 20, 2023
lostluck added a commit to lostluck/beam that referenced this pull request Feb 20, 2023
* [prism] internal/worker + tentative data

* [prism] B method comments.

* [prism] worker PR comments

---------

Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com>
@lostluck lostluck deleted the prism-worker branch February 21, 2023 05:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants