Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: lunchpail enqueue commands #122

Merged
merged 1 commit into from
Aug 12, 2024
Merged

feat: lunchpail enqueue commands #122

merged 1 commit into from
Aug 12, 2024

Conversation

starpit
Copy link
Member

@starpit starpit commented Aug 12, 2024

  • lunchpail enqueue file
  • lunchpail enqueue s3
  • lunchpail qdone -- for dispatchers to signal they have no more work to dispatch
  • Ports ParameterSweep and ProcessS3Objects to use these features.
  • Removes the ProcessS3Objects shell script. That dispatcher is now entirely go (though still with yaml wrappers that we should reevaluate).
  • TODO: the ParameterSweep shell script remains, though it does nowuse lunchpail enqueue file.

This helps us remove a few lingering reliances on rclone. It should also help with implementing custom dispatchers, as they should no longer need to deal with rclone and queue path conventions directly.

- `lunchpail enqueue file`
- `lunchpail enqueue s3`
- `lunchpail qdone` -- for dispatchers to signal they have no more work to dispatch

This helps us remove a few lingering reliances on rclone. It should also help with implementing custom dispatchers -- they should no longer need to deal with rclone and queue path conventions directly.

Ports ParameterSweep and ProcessS3Objects to use these features.

Signed-off-by: Nick Mitchell <nickm@us.ibm.com>
@starpit starpit merged commit 3608b66 into IBM:main Aug 12, 2024
6 checks passed
@starpit starpit deleted the no-rclone branch August 12, 2024 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant