Scheduler rate limit functionality

[jpbruinsslot]

Changes

This PR adds rate-limiting capabilities to the scheduler for tasks. Boefjes that are subject to rate limiting (defined in the boefje manifest) are being tracked by an in-memory rate limiter leveraging the package limits. When a task is subject to a rate-limit it will get the DELAYED status and will be postponed until it is allowed again. A thread will check for delayed tasks and put them on the queue when they're ready.

This will expect some changes in the katalogus mainly exposing a grouping/namespacing on which to rate limit and the parseable rate limit.

Issue link

Closes #1317

Processess

For a boefje scheduler when a task gets created it will get check whether the boefje is rate limited or not:

flowchart TD
    A[Task Creation] --> B{is_task_rate_limited}
    B -->|Yes| C[set Task to status <br>DELAYED]
    B ---->|No| E[End]

Loading

Checking if a task is rate limited:

graph TD
    A[Start] --> B{rate_limit is None?}
    B -- Yes --> C[Return False]
    B -- No --> D{Try to parse rate_limit}
    D --> E{parsed_rate_limit is None?}
    E -- Yes --> F[Raise ValueError]
    E -- No --> G[Lock rate_limiter]
    G --> H[Get identifier_template]
    H --> I[Render identifier from template]
    I --> J[Test rate_limiter with identifier]
    J --> K{can_consume?}
    K -- No --> L[Return True]
    K -- Yes --> M{hit?}
    M -- Yes --> N[Hit rate_limiter]
    N --> O[Return False]
    M -- No --> O
    D --> P[Caught ValueError]
    P --> Q[Log warning]
    Q --> R[Raise exc]

Loading

Within the boefje scheduler a new process (push_tasks_for_delayed_tasks) is running next to:

• check for scan profile changes
• check for enabled boefjes
• check for rescheduling (random ooi endpoint octopoes)

This push_tasks_for_delayed_tasks method gets all tasks that have the status of DELAYED. And will be checked whether or not task can be pushed onto the queue.

[jpbruinsslot]

@ammar92 I've implemented cross-organisational support would you mind taking a look?

[praseodym]

Originally posted at #1413 (review), but still valid:

As far as I understand, the rate limiter is a property of the BoefjeScheduler class. This class has a separate rate_limiter instance per organisation, which means that rate limiting only works per organisation.

In larger deployments, API keys such as for Shodan will likely be shared by multiple KAT organisations. How do we implement rate limiting for that scenario?

[jpbruinsslot]

Originally posted at #1413 (review), but still valid:

As far as I understand, the rate limiter is a property of the BoefjeScheduler class. This class has a separate rate_limiter instance per organisation, which means that rate limiting only works per organisation.

In larger deployments, API keys such as for Shodan will likely be shared by multiple KAT organisations. How do we implement rate limiting for that scenario?

I've updated the code so that the rate limiter memory is now on the application level. The identifier of the rate limiter can then be formatted with a key for instance so that it will be shared across multiple organisations.

[underdarknl]

In larger deployments, API keys such as for Shodan will likely be shared by multiple KAT organisations. How do we implement rate limiting for that scenario?

This also means that we should probably include both 'shodan' and the actual key in the 'group' to make sure we don't limit shodan requests across the whole install even though we might have multiple keys available.

[underdarknl]

I feel like using Jinja here is probably too much of a good thing. Woudn't fstring be more than capable enough?

[jpbruinsslot]

I think the main reason was to give more flexibility to the user to construct a rate limit identifier that can use the information embedded in the task.

@ammar92 am I correct in that assumption?