Configure modified retry for uploading partition yamls to GCS #1292
Labels
priority: p1
High priority. Fix may be included in the next release.
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
When uploading a large number of yamls to GCS (10,000) we are encountering a 503 error. This indicates the connection to GCS was interrupted, and is commonly encountered when uploading a large number of files via python.
The exact error message:
'Request failed with status code', 503, 'Expected one of', <HTTPStatus.OK: 200>
The recommendation with 5xx errors is to retry, and we have some sample code as a reference: https://github.com/googleapis/python-storage/blob/main/samples/snippets/storage_configure_retries.py#L55
The request is to please modify _write_gcs_file in gcs_helper.py to use retries.
Suggestion (based on above sample code):
Ideally, the values (deadline, initial, multiplier, maximum) can be parameterized, so that end users can modify the values to get optimal performance.
The text was updated successfully, but these errors were encountered: