-
-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NVD API returns transient 403 response with API key #6195
Comments
I can confirm the same issue with gradle. During initial NVD download, I get transient 403 errors. Here's the error:
|
Update, I suspect this is related to API throttling. ODC has a default delay in between requests but I suspect that isn't enough to keep us out of API throttling limit entirely. After parsing this carefully: https://nvd.nist.gov/general/news/API-Key-Announcement it sounds like the limit is applied based on an endpoint mapped to the key. I think this bit is important:
I read this as, depending on how your networking plumbing works and how that looks to NVD, they will aggregate all of your org's requests into a single quota. This means you could get throttled even though you're not sharing the key. |
I was thinking the same thing - this must one of NVD's methods to manage rate limiting (although 403 seems like the wrong initial response unless is was preceded by a 429 somewhere, but I'm not getting rate limiting messages from ODC). I keep a container image that's preloaded with the latest vulnerability updates for multiple users/projects. Previously, I was rebuilding the full database every 4 hours - easy, and fast, so no driver to make more complicated. Since this I've moved to a build that takes the vulnerability data from the previous image build and updates ODC with that. The key is obviously getting one good build, which was a challenge with over 230,000 records to retrieve, but since then the OCD updates have typically less than 50 records to update. |
I have run into this issue when running a build from a GitHub action, for example: https://github.com/spt-development/spt-development-cid-jms-spring/actions/runs/7071629550/job/19249535231 When running the action, because a new container is spun up each time the database is empty and therefore I get throttled as others have suggested. I know of no way of re-using containers from GitHub actions (I'm no expert) and therefore @aarongoldenthal 's solution wouldn't work for me. Depending on how NVD determines when to throttle, this problem could be exacerbated when running concurrent builds for other projects. I have worked around this for now, by not passing in the API key and accepting that the build will be slower for example: https://github.com/spt-development/spt-development-cid-jms-spring/actions/runs/7072135068/job/19250625512 It would seem straight forward to fix this problem in the maven plugin, if the API returned a more appropriate error code and/or something to indicate throttling (if that is the issue) in the response body. NOTE I have not run into this problem (yet) when running locally with a database that doesn't require completely re-populating. |
Setting (The default delay in the code is currently 2000ms, perhaps this should be changed). |
Ah increasing the delay as suggested by @gustafg looks as though it should help. I have also discovered this action to avoid rebuilding the database each time: https://github.com/marketplace/actions/dependency-check |
The action is currently only 8.4.3 - it will be upgraded soon. |
See #6204 - initial attempt at documenting strategies people use to cache the H2 database to improve execution time and avoid 403 errors. |
Are people seeing this only in environments with multiple builds that may be sharing the same NVD API key? |
I had difficulty with an API key that was not shared. |
@gustafg how recently? and with which version of ODC? |
@jeremylong I'm seeing as a single user, but I am running on gitlab.com shared CI runners. I last saw with ODC v9.0.2 on 12/01 19:10 CST. That's the last full build that I've done, since then starting from the cache and each build has no more than 150 NVD updates. |
Having the similar issue, using ODC v9.0.2. Using an api key throws the error below Not using a key seems to get better results. Get the update and check done |
Same error with 9.0.2 plugin ... nvdApiDelay configuration property does not seem to help. [ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
...
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 403
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:346)
...
57947 [ERROR] Failed to execute goal org.owasp:dependency-check-maven:9.0.2:check (default) on project Utils: Fatal exception(s) analyzing ArnesUtils: One or more exceptions occurred during analysis:
57947 [ERROR] UpdateException: Error updating the NVD Data
57947 [ERROR] caused by NvdApiException: NVD Returned Status Code: 403
57947 [ERROR] NoDataException: No documents exist |
You can check to ensure you API Key is valid: https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs
If this does not return JSON you likely need to request a new api key. |
I've been getting our projects up-to-date & using the latest version 9.0.2 Yet, in our jenkins instance (using the plugin), if I didnt set or add the delay flag we would always get a 403 error: (I tested values with 6000,5000,4000,3000) & landed on the default value. (all those values worked and we didnt get a 403 error anymore) Leave off the delay flag and it errors out
So maybe gradle is respecting the default delay without setting it, and the cli isn't? |
If you have multiple builds happening at the same time - using the same API key you could hit the NVD rate limiting threshold. Ideally, in an environment with multiple builds you would implement some sort of caching strategy. The documentation on this continues to evolve/improve: #6220 |
@jeremylong thank you! Got the daily catch strategy setup. Should have done this long ago. So much more efficient! |
I have the same issue with the version 9.0.7, with the maven plugin. the execution with the API-Key starts download but runs into a 403 error: if i run the same check without the API-Key it succeeds. |
- increase default delay from 2000 to 3500 - resolves #6195
@jeremylong thank you - increasing the nvdApiDelay from 2000 to 3500 solved the problem for me, waiting for the new version :-) |
Describe the bug
When using an API key, the NVD API has started returning a transient 403 response. It occurs in the middle of a database update, so not a key configuration issue (and when retrying the key is used successfully). Some ODC database updates do complete, but this has occurred about half of the time since upgrading to v9.0.2.
This is related to #6180 and #6149, but is still occurring with the 9.0.2 CLI.
Version of dependency-check used
The problem occurs using version 9.0.2 of the cli (from the
owasp/dependency-check
image)To Reproduce
Run
/usr/share/dependency-check/bin/dependency-check.sh --updateonly
, but since transient it's hard to reproduce reliably.Expected behavior
Database update should occur without error.
The text was updated successfully, but these errors were encountered: