Memory is critically overloaded WARNING

[stephenleo]

I'm running the quickstart code in WSL2 on Windows 11. Crawlee is not using the full available memory in WSL2. I have 15Gi total memory in WSL2

$  free -mh
               total        used        free      shared  buff/cache   available
Mem:            15Gi       2.7Gi       8.9Gi       3.0Mi       3.9Gi        12Gi
Swap:          4.0Gi          0B       4.0Gi

Crawlee is only using 3.88GB for some reason.

[crawlee._autoscaling.snapshotter] WARN  Memory is critically overloaded. Using 5.24 GB of 3.88 GB (135%). Consider increasing available memory.

How can I make Crawlee use a larger percentage of available memory?

Code (same code as the quickstart):

import asyncio

from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext


async def main() -> None:
    # PlaywrightCrawler crawls the web using a headless browser controlled by the Playwright library.
    crawler = PlaywrightCrawler()

    # Define a request handler to process each crawled page and attach it to the crawler using a decorator.
    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        # Extract relevant data from the page context.
        data = {
            'url': context.request.url,
            'title': await context.page.title(),
        }
        # Store the extracted data.
        await context.push_data(data)
        # Extract links from the current page and add them to the crawling queue.
        await context.enqueue_links()

    # Add first URL to the queue and start the crawl.
    await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

[janbuchar]

Hello, and thank you for your interest in Crawlee! By default, Crawlee uses 25% of the available memory. You can alter this by setting the CRAWLEE_AVAILABLE_MEMORY_RATIO to 0.75, for instance. Or you can pass a custom crawlee.Configuration object to your crawler, with an adjusted value of the available_memory_ratio attribute.

Alternatively, you can use CRAWLEE_MEMORY_MBYTES or memory_mbytes to set the memory limit directly.