Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-16100 test: Fix stopping daos_test during timeout #15275

Draft
wants to merge 25 commits into
base: master
Choose a base branch
from
Draft

DAOS-16100 test: Fix stopping daos_test during timeout #15275

wants to merge 25 commits into from

Conversation

phender
Copy link
Contributor

@phender phender commented Oct 8, 2024 •
edited
Loading

Fix stopping timed out processes run by a JobManager class by only
searching for and killing the command executable being run by clush,
orterun, mpirun, etc. Add a new harness/cmocka.py test to verify the
stopping of the processes with a test timeout.

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_daos_management

Required-githooks: true

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

@phender
DAOS-16100 test: Fix stopping daos_test during timeout
ac9ab31 
Properly dstop the daos_test process if the test encounters a timeout
while running.

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_daos_management

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@phender phender requested review from a team as code owners October 8, 2024 17:44
@phender phender marked this pull request as draft October 8, 2024 17:44
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 8, 2024

Ticket title is 'daos_test/suite.py:DaosCoreTest.test_daos_degraded_ec - cart_ctl error due to errored rank 0'
Status is 'In Progress'
Labels: '2.6.0rc1,ci_impact,pr_test,tcp_provider,testp2'
https://daosio.atlassian.net/browse/DAOS-16100

@phender
Merge branch 'master' into pahender/DAOS-16100
1129c26
@phender
Force a timeout for debug.
71a4142 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_daos_management

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/2/execution/node/938/log

@phender
Force test timeout
cb91989 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_daos_management

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@phender
Force timeout
6c05d56 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_daos_management

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/4/execution/node/938/log

@phender
Merge branch 'master' into pahender/DAOS-16100
9151aad
@phender
Updates.
bd8b1dd 
When stopping cmocka commands only use the executable name to find a
pkill match.

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: DaosCoreTestDfs DaosCoreTestDfuse harness_cmocka test_daos_management

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/5/execution/node/800/log

@phender
Updates.
456f256 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: DaosCoreTestDfs DaosCoreTestDfuse harness_cmocka test_daos_management
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@phender
Cleanup.
c366c64 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: DaosCoreTestDfs DaosCoreTestDfuse harness_cmocka test_daos_management
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/7/execution/node/826/log

@phender
Ensure registered tearDown steps are run in TestWithoutServers
e496214 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: DaosCoreTestDfs DaosCoreTestDfuse harness_cmocka test_daos_management MultiEnginesPerSocketTest FaultDomain
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@phender
Add missing modified file.
9cada0e 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: DaosCoreTestDfs DaosCoreTestDfuse harness_cmocka test_daos_management MultiEnginesPerSocketTest FaultDomain
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/9/execution/node/826/log

@phender
Define hosts for cleanup.
0c3493f 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: DaosCoreTestDfs DaosCoreTestDfuse harness_cmocka test_daos_management MultiEnginesPerSocketTest FaultDomain
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/10/execution/node/825/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/10/execution/node/1063/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/10/execution/node/1126/log

@phender
Updates.
34fc71b 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: DaosCoreTestDfs DaosCoreTestDfuse harness_cmocka test_daos_management MultiEnginesPerSocketTest FaultDomain
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@phender
Fix assign hosts.
4b58ad7 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: DaosCoreTestDfs DaosCoreTestDfuse harness_cmocka test_daos_management MultiEnginesPerSocketTest FaultDomain
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15275/11/display/redirect

@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/12/execution/node/826/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/12/execution/node/1034/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/12/execution/node/1080/log

@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/13/execution/node/825/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/13/execution/node/1063/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/13/execution/node/1079/log

@phender
Merge branch 'master' into pahender/DAOS-16100
f5244e9
@phender
Don't stop the systemctl command.
04594cb 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: DaosCoreTestDfs DaosCoreTestDfuse harness_cmocka test_daos_management MultiEnginesPerSocketTest FaultDomain
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/14/execution/node/825/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15275/14/display/redirect

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15275/14/display/redirect

@phender
Fix basic tests.
cd45c23 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: DaosCoreTestDfs DaosCoreTestDfuse harness_cmocka test_daos_management MultiEnginesPerSocketTest FaultDomain
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/15/execution/node/825/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15275/15/display/redirect

@phender
Don't kill orterun/mpirun commands.
12cd18a 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: DaosCoreTestDfs DaosCoreTestDfuse harness_cmocka test_daos_management MultiEnginesPerSocketTest FaultDomain
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/16/execution/node/825/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15275/16/display/redirect

@phender
Cleanup.
f76b169 
Updated description:

Fix stopping timed out processes run by a JobManager class by only
searching for and killing the command executable being run by clush,
orterun, mpirun, etc. Add a new harness/cmocka.py test to verify the
stopping of the processes with a test timeout.

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: pr daos_test dfuse_test test_load_mpi HarnessCmockaTest
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@phender phender marked this pull request as ready for review November 2, 2024 05:28
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/17/execution/node/825/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15275/17/display/redirect

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15275/17/display/redirect

@phender phender marked this pull request as draft November 4, 2024 16:47
@phender
Merge branch 'master' into pahender/DAOS-16100
b3fa745
@phender
Merge branch 'master' into pahender/DAOS-16100
6e4e1b2
@phender
Add check for stopping pid 1.
6d17d3d 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: pr daos_test dfuse_test test_load_mpi HarnessCmockaTest
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/18/execution/node/825/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/18/execution/node/1033/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/18/execution/node/1087/log

@phender
Merge branch 'master' into pahender/DAOS-16100
b959c10
@phender
Only match exact full commands if requested in stop_processes.
a6ff5d2 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: pr daos_test dfuse_test test_load_mpi HarnessCmockaTest
Allow-unstable-test: true

Required-githooks: true

Signed-off-by: Phil Henderson <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15275/19/execution/node/825/log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daltonbohning daltonbohning Awaiting requested review from daltonbohning

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@phender @daosbuild1