Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider adding retry to some core library functions #648

Open
stucka opened this issue May 20, 2024 · 0 comments
Open

Consider adding retry to some core library functions #648

stucka opened this issue May 20, 2024 · 0 comments

Comments

@stucka
Copy link
Contributor

stucka commented May 20, 2024

I hadn't seen this kind of failure before, but it might be easy enough to add another retry decorator:

2024-05-20 19:26:01,222 - warn.utils - Requesting https://dlt.ri.gov//media/15796/download?language=en
2024-05-20 19:26:05,656 - warn.utils - Response code: 200
2024-05-20 19:26:05,657 - warn.cache - Writing to data/warn-scraper/cache/ri/WARN Report.xlsx
Traceback (most recent call last):
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/urllib3/response.py", line 737, in _error_catcher
yield
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/urllib3/response.py", line 883, in _raw_read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(29126 bytes read, 136 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/requests/models.py", line 816, in generate
yield from self.raw.stream(chunk_size, decode_content=True)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/urllib3/response.py", line 1043, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/urllib3/response.py", line 963, in read
data = self._raw_read(amt)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/urllib3/response.py", line 891, in _raw_read
self._fp.close()
File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/contextlib.py", line 137, in exit
self.gen.throw(typ, value, traceback)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/urllib3/response.py", line 761, in _error_catcher
raise ProtocolError(arg, e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(29126 bytes read, 136 more expected)', IncompleteRead(29126 bytes read, 136 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/warn/cli.py", line 79, in
main()
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/warn/cli.py", line 75, in main
runner.scrape(scrape)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/warn/runner.py", line 52, in scrape
data_path = state_mod.scrape(self.data_dir, self.cache_dir)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/warn/scrapers/ri.py", line 53, in scrape
excel_path = cache.download(f"{state_code}/WARN Report.xlsx", excel_url)
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/warn/cache.py", line 105, in download
for chunk in r.iter_content(chunk_size=8192):
File "/home/runner/.local/share/virtualenvs/warn-github-flow-R1xICqqL/lib/python3.9/site-packages/requests/models.py", line 818, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(29126 bytes read, 136 more expected)', IncompleteRead(29126 bytes read, 136 more expected))
make: *** [Makefile:71: scrape] Error 1
Error: Process completed with exit code 2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant