Too many threads, CPU use

[babbottscott]

I seem to remember a configurable thread limit per ultragrep process. This appears to be no longer the case. cpu use is too high on logs servers with a thread per log file searched. There needs to be some cap.

[osheroff]

maybe tricky to pull off given the architecture of a single C process per log file grepped. We could try rate limiting by forcing ug_guts to block writing to its pipe, but to do it "right" would probably mean a significant re-architecture, making ug_guts a multi-threaded process.

Nothing we could do with nice-levels?

[babbottscott]

It's already renicing to +19. There was a suggestion to use taskset to add affinity to a subset of cpus. Perhaps that would be a cleaner implementation?

[osheroff]

yeah, that could be fine. How many cpus need we reserve?

[babbottscott]

First take I'd say 8 out of the 24 at most, but I'd love to see it configurable. Taskset takes strings like '1-4,6,8' for specifying affinity. If we could specify that string in the config, that would be great.

[briser]

dusts off thread Bump.