update_cp_org_libraries.py runs veryyyy slow #263
Comments
|
We are now often seeing the job exceed the 6 hour job runtime limit. For example: https://github.com/adafruit/circuitpython-org/actions/runs/6809744073/job/18516615845. There is some GitHub API rate limiting going on too (see in that job), but it would be really nice to make this run faster. |
|
Thinking about your local test in #361, are we doing the queries without credentials? If we did them with credentials, would we avoid the rate limit? |
|
I'm not sure about authorization. I filed #348 thinking that this process was using Classic PATs (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) which are deprecated or discouraged; I forget the exact context that led to me filing that issue but it seems to be related to this note in the docs:
Using PATs of some kind seems to be the Proper Way (TM) to do this, but "fine-grained" PATs are the preferred way now, if we don't want to deal with having a GitHub App: https://docs.github.com/en/actions/security-guides/automatic-token-authentication#granting-additional-permissions I am pretty sure SOME kind of authentication/token is at play here, otherwise the unauthenticated limit is something like 20 API calls an hour, far too few. |
|
My local test was without authorization, but that was kinda-nice since it let me easily hit the timeout case. For the run in circuitpython-org that's troubling us, we appear to be using a token via ADABOT_GITHUB_USER and _ACCESS_TOKEN: since the value of repository secrets can't be inspected, I don't know what user & token is in play. |
|
I think these tokens are OK, because I changed them recently, carefully, and other Actions jobs depend on them. |
|
A successful run looks like this: so we have:
A failed run: here,
Besides adding debugging another idea is to prepend the command with |
|
also we could sleep +60 seconds after we think the rate limit will reset, not +1 second; since we hit the rate limit after 40-45 minutes and it resets after 1 hour, this doesn't really lower our throughput much. |
|
Can we insert delays between our requests? Would that throttle it enough to avoid the rate limit? |
) The "core rate limit reset" time is in UTC, while github actions may be running in another timezone, such as PST. The below example was run in my local timezone, CST: ```python >>> import os, datetime >>> import github as pygithub >>> GH_INTERFACE = pygithub.Github(os.environ.get("ADABOT_GITHUB_ACCESS_TOKEN")) >>> GH_INTERFACE.get_rate_limit().core.reset - datetime.datetime.now() datetime.timedelta(seconds=24750, microseconds=356986) >>> GH_INTERFACE.get_rate_limit().core.reset - datetime.datetime.utcnow() datetime.timedelta(seconds=3147, microseconds=87271) ``` Use utcnow() instead of now() so that the sleep is for the correct duration. (utcnow is deprecated, but the correct alternative, `now(tz=datetime.timezone.utc)` does not work: `GH_INTERFACE.get_rate_limit().core.reset` is a naive timestamp and can't be compared with a tz-aware object)
|
I think I found the real problem 🎉 |
|
Fixed by #362. |
|
I saw another failed 6-hour run, despite circuitpython-org being updated to include #362. 😕 |
|
oh drat |
|
I have thought about moving part of the CI into libraries themselves somehow. The basic idea is to have repos trigger when updated (or scheduled) to push a record of the information gathered either to a central repo (like a folder in this repository, for example) or possibly an S3 bucket. This has the advantage of spreading out or eliminating API calls, as well as distributing the burden over multiple CI runs across the libraries. This would allow this CI to just collect them all. |
Results of my initial tests of the functions that get run.
Total time per repo:
3.5svalidate_actions_state:
0.459673svalidate_contents:
0.433544svalidate_core_driver_page:
0.003178svalidate_default_branch:
0.000001svalidate_in_pypi:
0.040591svalidate_labels:
0.117371svalidate_passes_linting:
0.639910svalidate_readthedocs:
1.361857svalidate_release_state:
0.333909svalidate_repo_state:
0.134701sSo the real big one is validate_readthedocs. That one is definitely one of the more complicated ones but I think we could for sure simplify it.
validate_passes_linting also takes a while but it actually clones the repository so that make sense.
The text was updated successfully, but these errors were encountered: