Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](routine load) delay schedule EOF tasks to avoid too many small transactions (#39975) #40498

Merged
merged 1 commit into from
Sep 8, 2024
Merged

[improve](routine load) delay schedule EOF tasks to avoid too many small transactions (#39975) #40498

merged 1 commit into from
Sep 8, 2024

Commits on Sep 7, 2024

  1. [improve](routine load) delay schedule EOF tasks to avoid too many sm… 

    …all transactions (apache#39975)

We encountered a scenario where a large number of small transactions
were generated, resulting in an impact on query performance:
Kafka's data comes in batches of very small data every very short time,
which leads to tasks being frequently scheduled and ending very quickly,
resulting in a large number of small transactions.

To solve this problem, we delay the scheduling of tasks that perceive
EOF, which would not delay data consumption, for perceiving EOF
indicates that the consumption speed is greater than the production
speed.
    @sollhui
    sollhui committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    b665ddf View commit details
    Browse the repository at this point in the history