How to stop crawling a web site when a goal is reached

[sebpiq]

I have a list of web sites from which I am trying to scrape a given piece of info. For each site, once I have found that info, I want to stop and move on to the next (with several sites being scraped concurrently).

I have tried the following approach (emptying the request queue when my goal is found) :

request_queue = await RequestQueue.open()
crawler = PlaywrightCrawler(
    request_provider=request_queue,
    headless=True,  # Show the browser window.
    browser_type='firefox',  # Use the Firefox browser.
)
    
await crawler.add_requests([root_url])

@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
    # ...
    if found:
        await request_queue.drop()

But that's actually raising an error :

 ValueError: Request queue with id "default" does not exist.

Any idea how I should proceed to have a finer control over the request queue ? Thanks !

[sebpiq]

Also, should RequestQueue have a delete_request function to allow removing requests from the queue when they're not needed anymore?

[janbuchar]

The short answer is that the typical use case does not require this.

[janbuchar]

Let me get the requirements straight. There are several websites, and you want to crawl each of them until you find something specific. Do you use the same request handler for each page? I'm asking because it would make a lot of sense to keep the same crawler instance so that parallelism works well.

[sebpiq]

it would make a lot of sense to keep the same crawler instance so that parallelism works well

Yes, I was thinking also that it would make sense to use the same crawler.

There are several websites, and you want to crawl each of them until you find something specific

Yes that's it. In each of these websites, I am looking for a legal document. Once I have found it I can just move on. But so yes, this is the exact same handler for each url.

[janbuchar]

So, a simple solution that might be good enough would be to keep a set of processed websites outside of the crawler and then do something like this in your request handler:

from urllib.parse import urlparse

finished_hostnames = set[str]()
crawler = PlaywrightCrawler()

@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
  hostname = urlparse(context.request.url).hostname
  if hostname in finished_hostnames:
    return

  # ... keep crawling ...

  if found:
    finished_hostnames.add(hostname)
  

Of course, pages from finished websites that were enqueued before you marked them as finished will be opened anyway, but at least the crawler won't go deeper from them.

[EngineerKhan]

but at least the crawler won't go deeper from them.

I have a bit similar question: is there a way to specify the maximum depth to crawl in? (I am using BeautifulSoupCrawler).

[janbuchar]

@EngineerKhan I don't see how this is similar, but see #460