Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Disk Manager]: Move hanging tasks metric to collect lister metrics #1916

Merged
merged 84 commits into from
Sep 10, 2024
Merged

[Disk Manager]: Move hanging tasks metric to collect lister metrics #1916

merged 84 commits into from
Sep 10, 2024

Commits on Aug 29, 2024

  1. Do not report hanging tasks

    @jkuradobery
    jkuradobery committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    34c0c93 View commit details
    Browse the repository at this point in the history

  2. Use estimated duration * 2 as traceback printing deadline

    @jkuradobery
    jkuradobery committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    ae0b097 View commit details
    Browse the repository at this point in the history

  3. Add IntervalValue to persistence/ydb 

    We need this to use time.Duration in YQL queries
    @jkuradobery
    jkuradobery committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    4d7094a View commit details
    Browse the repository at this point in the history

  4. Implement hanging tasks listing

    @jkuradobery
    jkuradobery committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    14e4749 View commit details
    Browse the repository at this point in the history

  5. Fix runner_metrics compilation

    @jkuradobery
    jkuradobery committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    012a1f9 View commit details
    Browse the repository at this point in the history

  6. Add hanging tasks collection to collect lister metrics task

    @jkuradobery
    jkuradobery committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    211534a View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. Count waiting to run/cancel tasks as hanging

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    c4d0022 View commit details
    Browse the repository at this point in the history

  2. Add unit-test for ListTasksHanging

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    9c3797d View commit details
    Browse the repository at this point in the history

  3. Fix test build

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    95bdc6d View commit details
    Browse the repository at this point in the history

  4. Fix sql syntax

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    c4c92c1 View commit details
    Browse the repository at this point in the history

  5. Add limit to sql query

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    f7bd7fd View commit details
    Browse the repository at this point in the history

  6. Remove requests to missing tables

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    56a2d4f View commit details
    Browse the repository at this point in the history

  7. Remove waiting statuses from the test

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    23adb6f View commit details
    Browse the repository at this point in the history

  8. Log hanging tasks in the test

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    dcc4461 View commit details
    Browse the repository at this point in the history

  9. Log hanging tasks creation in the test

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    7b99c72 View commit details
    Browse the repository at this point in the history

  10. Fix hanging tasks query

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    0feefee View commit details
    Browse the repository at this point in the history

  11. Report task id to monitoring

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    e69474c View commit details
    Browse the repository at this point in the history

  12. Add MaxReportHangingTaskID option

    @jkuradobery
    jkuradobery committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    6294cfb View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. Fix runner_test.go

    @jkuradobery
    jkuradobery committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    52e2102 View commit details
    Browse the repository at this point in the history

  2. Fix runner_test.go

    @jkuradobery
    jkuradobery committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    cdddefc View commit details
    Browse the repository at this point in the history

  3. Use UNION ALL instead of the UNION to be compatible with older YDB ve… 

    …rsions
    @jkuradobery
    jkuradobery committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    b948155 View commit details
    Browse the repository at this point in the history

  4. Set all hanging tasks gauges by type to zero if no hanging tasks present

    @jkuradobery
    jkuradobery committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    af5fc73 View commit details
    Browse the repository at this point in the history

  5. Change all receivers for collectListerMetricsTask to pointer receivers

    @jkuradobery
    jkuradobery committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    8285e0f View commit details
    Browse the repository at this point in the history

  6. Fix code style

    @jkuradobery
    jkuradobery committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    8169fc6 View commit details
    Browse the repository at this point in the history

  7. Fix code style

    @jkuradobery
    jkuradobery committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    951d0d1 View commit details
    Browse the repository at this point in the history

  8. Fix unhandled error in hanging task duration parsing

    @jkuradobery
    jkuradobery committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    8cebd4a View commit details
    Browse the repository at this point in the history

  9. Fix style

    @jkuradobery
    jkuradobery committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    1381f34 View commit details
    Browse the repository at this point in the history

  10. Fix style

    @jkuradobery
    jkuradobery committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    0ee931a View commit details
    Browse the repository at this point in the history

  11. Fix (some) review discussions

    @jkuradobery
    jkuradobery committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    4f79b15 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Fix discussions

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    d91a768 View commit details
    Browse the repository at this point in the history

  2. Add separate metrics registry for scheduler in task_test

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    8e6b22b View commit details
    Browse the repository at this point in the history

  3. Introduce test for hanging tasks metrics

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    f14b2ed View commit details
    Browse the repository at this point in the history

  4. Adjust sleep interval in the hanging tasks test

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    981289a View commit details
    Browse the repository at this point in the history

  5. Fix missing idempotency key for task

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    e52e9b8 View commit details
    Browse the repository at this point in the history

  6. Fix missing idempotency key for task

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    ce09918 View commit details
    Browse the repository at this point in the history

  7. Add waiting for task ended, remove assert for cancelling

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    d9f1521 View commit details
    Browse the repository at this point in the history

  8. Add logging

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    f133de0 View commit details
    Browse the repository at this point in the history

  9. Increase task waiting timeout for hanging tasks test

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    42ffd5f View commit details
    Browse the repository at this point in the history

  10. Add additional logging

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    df41bf8 View commit details
    Browse the repository at this point in the history

  11. Increase runners count in tasks test

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    afe4a08 View commit details
    Browse the repository at this point in the history

  12. Fix test

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    40f688d View commit details
    Browse the repository at this point in the history

  13. Increase number of runners

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    3177c6d View commit details
    Browse the repository at this point in the history

  14. Add more logging

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    a15238d View commit details
    Browse the repository at this point in the history

  15. Fix test, remove unnecessary logging

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    09b8d67 View commit details
    Browse the repository at this point in the history

  16. Add return check for all mocks

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    5fdb5a6 View commit details
    Browse the repository at this point in the history

  17. Add missing mocks

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    7fd6bfc View commit details
    Browse the repository at this point in the history

  18. Add sleep in the end of the test

    @jkuradobery
    jkuradobery committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    e881555 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Simplify ExecutionContext.GetDeadline() 

    Pass hangingTaskTimeout through the runner to execution context.
Use ExecutionContext interface instead of the concrete class in runnerMetrics.
    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    a2696ac View commit details
    Browse the repository at this point in the history

  2. Fix typo in scheduleHangingTask name

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    af7c559 View commit details
    Browse the repository at this point in the history

  3. rename defaultHangingTasksTimeout -> hangingTaskTimeout

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    031fff5 View commit details
    Browse the repository at this point in the history

  4. rename exceptHangingTaskTypes -> exceptTaskTypes

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    9256824 View commit details
    Browse the repository at this point in the history

  5. Fix logs in collect lister metrics task

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    65c33f2 View commit details
    Browse the repository at this point in the history

  6. Refactor TestHangingTasksMetrics

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    6850f44 View commit details
    Browse the repository at this point in the history

  7. Add estimateMissMultiplier as a parameter

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    f7ab282 View commit details
    Browse the repository at this point in the history

  8. Use max(default estimate, actual estimate) for hanging tasks

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    bd1ad12 View commit details
    Browse the repository at this point in the history

  9. Fix test

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    cca7c65 View commit details
    Browse the repository at this point in the history

  10. Fix paddings

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    576f8a2 View commit details
    Browse the repository at this point in the history

  11. Remove miscellaneous gauges

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    1de632b View commit details
    Browse the repository at this point in the history

  12. Fix formatting

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    5e9e10d View commit details
    Browse the repository at this point in the history

  13. Fix TestHangingTasksMetrics potential flaps

    @jkuradobery
    jkuradobery committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    585c6fe View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Fix review discussions

    @jkuradobery
    jkuradobery committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    95c0d54 View commit details
    Browse the repository at this point in the history

  2. Move hangingTaskTimeout, missedEstimatesUntilHanging to storage ydb

    @jkuradobery
    jkuradobery committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    1b0fd2b View commit details
    Browse the repository at this point in the history

  3. Rename GetEstimateMissMultiplier -> GetMissedEstimatesUntilHanging

    @jkuradobery
    jkuradobery committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    d10fc17 View commit details
    Browse the repository at this point in the history

  4. Fix incorrect query

    @jkuradobery
    jkuradobery committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    0d4728e View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Fix review discussions

    @jkuradobery
    jkuradobery committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    173c687 View commit details
    Browse the repository at this point in the history

  2. Fix storage ydb test

    @jkuradobery
    jkuradobery committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    857a35f View commit details
    Browse the repository at this point in the history

  3. Add two new tests for hanging tasks

    @jkuradobery
    jkuradobery committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    92d863f View commit details
    Browse the repository at this point in the history

  4. Fix test

    @jkuradobery
    jkuradobery committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    6d205c9 View commit details
    Browse the repository at this point in the history

  5. Simplify metrics test

    @jkuradobery
    jkuradobery committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    69bb602 View commit details
    Browse the repository at this point in the history

  6. Remove unused ydb test

    @jkuradobery
    jkuradobery committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    a38910e View commit details
    Browse the repository at this point in the history

  7. Do not report hanging tasks with id=all

    @jkuradobery
    jkuradobery committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    dd742cd View commit details
    Browse the repository at this point in the history

  8. Fix compilation

    @jkuradobery
    jkuradobery committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    b9b157e View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Fix review discussion

    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    a5d7bb9 View commit details
    Browse the repository at this point in the history

  2. Inline hangingTaskType in hanging tasks metric test

    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    952fc9e View commit details
    Browse the repository at this point in the history

  3. Fix formatting

    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    1a84652 View commit details
    Browse the repository at this point in the history

  4. Inline sensorName variable in collect_lister_metrics_task.go

    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    1f03d61 View commit details
    Browse the repository at this point in the history

  5. Inline setupExpectationsForHangingTasksMetricsTest 

    Wait for the first metric to appear in TestHangingTasksMetrics
    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    7ca89ca View commit details
    Browse the repository at this point in the history

  6. Rename taskId -> taskID

    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    20fd023 View commit details
    Browse the repository at this point in the history

  7. Rename hangingTaskTestContext -> hangingTaskTestFixture 

    Rename local variables accordingly. Simplify close().
    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    316d215 View commit details
    Browse the repository at this point in the history

  8. Rename hangingTaskTestFixture.close() -> teardown()

    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    73a3cc7 View commit details
    Browse the repository at this point in the history

  9. Rename config parameter accordingly

    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    70c03d9 View commit details
    Browse the repository at this point in the history

  10. Rename waitgroups in the test

    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    b16adcf View commit details
    Browse the repository at this point in the history

  11. Rename variables

    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    22631b8 View commit details
    Browse the repository at this point in the history

  12. Fix comments style

    @jkuradobery
    jkuradobery committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    5e0026d View commit details
    Browse the repository at this point in the history