Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logging to BigQuery can fail if number of rows to insert is too large #3008

Open
fdmalone opened this issue Sep 5, 2024 · 4 comments
Open
Labels
fixed; not released issues that have been fixed on the develop branch but have not yet been part of a tagged release.

Comments

@fdmalone
Copy link
Contributor

fdmalone commented Sep 5, 2024

Pulled from #2989. I ran into an issue where load_bq.py would fail as the number of rows was too large for insert_rows. My workaround was to batch over slices of the jobs list with a batch size set to 10000 (which I think is the upper limit for insert_rows).

I can send a PR with the patch but I was a bit hesitant given the lack of unit tests for the script and adding those would be a bit of work (faking the client etc)

@fdmalone fdmalone changed the title Logging to BigQuery can fail if number of rows to insert is to large Logging to BigQuery can fail if number of rows to insert is too large Sep 5, 2024
@mr0re1
Copy link
Collaborator

mr0re1 commented Sep 10, 2024

@fdmalone , your contribution would be very welcome, with or without tests.

@cdunbar13
Copy link
Contributor

Checking in to see if you've had a chance to submit a PR for this.

@fdmalone
Copy link
Contributor Author

I haven't, I'll try to get to it shortly

@tpdownes tpdownes added the fixed; not released issues that have been fixed on the develop branch but have not yet been part of a tagged release. label Oct 8, 2024
@tpdownes
Copy link
Member

tpdownes commented Oct 8, 2024

Marking this issue as "fixed, not released" because it is on our develop branch. It should be in the next release, which will probably be numbered v1.41.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fixed; not released issues that have been fixed on the develop branch but have not yet been part of a tagged release.
Projects
None yet
Development

No branches or pull requests

4 participants