Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change PeriodicSequence to report backlog accurately #32505

Merged
merged 2 commits into from
Sep 20, 2024

Conversation

scwhittle
Copy link
Contributor

Instead of reporting all the work it will do as backlog it should report just the scheduled based upon the current time that have not been produced yet.

This is done via GetSize but could be done by changing how Progress is reported. This seems more targeted fix as I'm not sure of all the implications of modifying progress.

Also fixup BOUNDED/UNBOUNDED since an effective end-of-time timestamp can be set.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@github-actions github-actions bot added the java label Sep 19, 2024
@scwhittle
Copy link
Contributor Author

scwhittle commented Sep 19, 2024

java sdk changes for #32506

Copy link
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@scwhittle
Copy link
Contributor Author

Verified with manual pipeline runs against Dataflow Runner that downscaling of pipeline with PeriodicSequence works with this change

Copy link
Contributor

@Abacn Abacn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! Python SDK has the same implementation, a follow up could be applying the same fix there

@scwhittle
Copy link
Contributor Author

Run Java_GCP_IO_Direct PreCommit

@scwhittle
Copy link
Contributor Author

Failure appears unrelated

@scwhittle scwhittle merged commit e1e6bdb into apache:master Sep 20, 2024
27 of 29 checks passed
@scwhittle scwhittle deleted the periodic_sequence branch September 20, 2024 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants