-
Notifications
You must be signed in to change notification settings - Fork 643
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: Add options for custom HTTP error status codes
This commit introduces two new optional properties to `CheerioCrawler` and `HttpCrawler`, allowing for finer control over how HTTP error status codes are handled: 1. `ignoreHttpErrorStatusCodes`: An array of HTTP response status codes that should be excluded from being considered as errors. By default, error consideration is triggered for status codes >= 500. 2. `additionalHttpErrorStatusCodes`: An array of extra HTTP response status codes that should be treated as errors. By default, error consideration is triggered for status codes >= 500. These options provide flexibility in specifying which HTTP response codes should be treated as errors and should be ignored during the crawling process.
- Loading branch information
1 parent
b389ccd
commit 083a162
Showing
3 changed files
with
120 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters