Introducing granular command timeouts global setting #9659

harikrishna-patnala · 2024-09-09T12:58:09Z

Description

CloudStack has a global setting "wait" which controls the wait time by the management server for all the commands which it sends. There are also some command specific timeout settings which controls specific commands like "ready.command.wait", "kvm.storage.online.migration.wait", etc.

This PR simplifies these configurations by introducing a new parent global setting which serves as a single point to configure any command timeout. The new global setting is "commands. timeout", this timeout overrides the wait global config. This holds a comma separated key value pairs containing timeout for specific commands. For example: DhcpEntryCommand=600, SavePasswordCommand=300, VmDataCommand=300.

There are also some hardcoded values defined in the code base for few commands. so the order of precedence of all these timeouts will be

"commands.timeout" > Existing global settings > Hardcoded value for the command > global "wait"

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

harikrishna-patnala · 2024-09-09T12:58:38Z

@blueorangutan package

blueorangutan · 2024-09-09T13:00:03Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2024-09-09T13:05:23Z

Codecov Report

Attention: Patch coverage is 45.45455% with 54 lines in your changes missing coverage. Please review.

Project coverage is 15.82%. Comparing base (15e25bc) to head (18890a7).
Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
...java/com/cloud/agent/manager/AgentManagerImpl.java	23.21%	42 Missing and 1 partial ⚠️
.../cloud/configuration/ConfigurationManagerImpl.java	73.80%	7 Missing and 4 partials ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##               main    #9659     +/-   ##
===========================================
  Coverage     15.81%   15.82%             
- Complexity    12554    12564     +10     
===========================================
  Files          5629     5629             
  Lines        492023   492141    +118     
  Branches      62519    63935   +1416     
===========================================
+ Hits          77813    77868     +55     
- Misses       405887   405948     +61     
- Partials       8323     8325      +2

Flag	Coverage Δ
uitests	`4.48% <ø> (-0.01%)`	⬇️
unittests	`16.61% <45.45%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

blueorangutan · 2024-09-09T13:49:11Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 11058

harikrishna-patnala · 2024-09-10T05:53:38Z

@blueorangutan package

blueorangutan · 2024-09-10T05:54:03Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-09-10T06:53:06Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11069

sureshanaparti

clgtm

sureshanaparti · 2024-09-10T08:48:35Z

@blueorangutan test

blueorangutan · 2024-09-10T08:50:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-09-11T00:10:06Z

[SF] Trillian test result (tid-11439)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52865 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9659-t11439-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_UpdateConfigParamWithScope	`Error`	0.08	test_global_settings.py
test_01_snapshot_usage	`Error`	4.24	test_usage.py

harikrishna-patnala · 2024-09-11T05:55:56Z

@blueorangutan package

blueorangutan · 2024-09-11T05:58:03Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

harikrishna-patnala · 2024-09-11T06:47:40Z

@blueorangutan package

blueorangutan · 2024-09-11T06:48:03Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-09-11T08:04:45Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11092

sureshanaparti · 2024-09-12T05:02:35Z

@blueorangutan test

blueorangutan · 2024-09-12T05:04:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-09-12T21:38:46Z

[SF] Trillian test result (tid-11457)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 57345 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9659-t11457-kvm-ol8.zip
Smoke tests completed. 138 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_UpdateConfigParamWithScope	`Error`	0.08	test_global_settings.py
test_01_snapshot_usage	`Error`	6.26	test_usage.py
test_02_unsecure_vm_migration	`Error`	497.36	test_vm_life_cycle.py

harikrishna-patnala · 2024-09-13T05:59:14Z

@blueorangutan package

blueorangutan · 2024-09-13T06:00:04Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-09-13T06:43:54Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 11107

harikrishna-patnala · 2024-09-13T10:19:42Z

@blueorangutan test

blueorangutan · 2024-09-13T10:22:03Z

@harikrishna-patnala a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-09-14T03:58:38Z

[SF] Trillian test result (tid-11470)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 61057 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9659-t11470-kvm-ol8.zip
Smoke tests completed. 138 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_snapshot_usage	`Error`	6.28	test_usage.py
test_02_unsecure_vm_migration	`Error`	448.61	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	259.21	test_vm_life_cycle.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	623.62	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Failure`	557.63	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	557.65	test_vpc_redundant.py

DaanHoogland · 2024-09-16T08:09:24Z

engine/components-api/src/main/java/com/cloud/agent/AgentManager.java

@@ -50,6 +50,10 @@ public interface AgentManager {
    ConfigKey<Integer> ReadyCommandWait = new ConfigKey<Integer>("Advanced", Integer.class, "ready.command.wait",
            "60", "Time in seconds to wait for Ready command to return", true);

+    ConfigKey<String> GranularWaitTimeForCommands = new ConfigKey<>("Advanced", String.class, "commands.timeout", "",
+            "This timeout overrides the wait global config. This holds a comma separated key value pairs containing timeout (in seconds) for specific commands. " +
+                    "For example: DhcpEntryCommand=600, SavePasswordCommand=300, VmDataCommand=300", true);


I would not make this one dynamic as the whole array needs to be loaded on every command sent to every agent. this will strain the system. The array should be constructed on load and be in memory readily available from then on. I.E.

Suggested change

"For example: DhcpEntryCommand=600, SavePasswordCommand=300, VmDataCommand=300", true);

"For example: DhcpEntryCommand=600, SavePasswordCommand=300, VmDataCommand=300", false);

Agreed @DaanHoogland, modified the code accordingly

harikrishna-patnala · 2024-09-18T12:04:21Z

@blueorangutan package

blueorangutan · 2024-09-18T12:06:03Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-09-18T13:17:26Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11146

harikrishna-patnala · 2024-09-19T03:41:06Z

@blueorangutan test

blueorangutan · 2024-09-19T03:42:04Z

@harikrishna-patnala a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-09-20T00:19:08Z

[SF] Trillian test result (tid-11518)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 71888 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9659-t11518-kvm-ol8.zip
Smoke tests completed. 138 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_snapshot_usage	`Error`	4.24	test_usage.py
test_01_secure_vm_migration	`Error`	135.20	test_vm_life_cycle.py
test_01_secure_vm_migration	`Error`	135.21	test_vm_life_cycle.py
ContextSuite context=TestVnfTemplates>:setup	`Error`	0.00	test_vnf_templates.py

…ep the backward compatibility

harikrishna-patnala · 2024-09-20T11:53:38Z

@blueorangutan package

blueorangutan · 2024-09-20T11:54:03Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-09-20T13:01:28Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11162

harikrishna-patnala · 2024-09-20T14:41:23Z

@blueorangutan test

blueorangutan · 2024-09-20T14:42:03Z

@harikrishna-patnala a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

boring-cyborg bot added component:agent component:integration-test component:orchestration Python Warning... Python code Ahead! labels Sep 9, 2024

harikrishna-patnala force-pushed the granularCommandTimeout branch from a666303 to f603974 Compare September 10, 2024 05:53

sureshanaparti approved these changes Sep 10, 2024

View reviewed changes

harikrishna-patnala added 3 commits September 11, 2024 11:23

Introducing granular command timeouts global setting

d72b4ce

fix marvin tests

f881ab5

Fixed log messages

76abf5d

harikrishna-patnala force-pushed the granularCommandTimeout branch from 58b66f6 to 76abf5d Compare September 11, 2024 05:55

some more log message fix

1a76b72

harikrishna-patnala force-pushed the granularCommandTimeout branch from 8d416b6 to 1a76b72 Compare September 11, 2024 06:46

rohityadavcloud requested a review from DaanHoogland September 12, 2024 06:38

Fix empty value setting

7f8f927

DaanHoogland reviewed Sep 16, 2024

View reviewed changes

Converted the global setting to non-dynamic

1de0cda

set wait on command only when granular wait is defined. This is to ke…

18890a7

…ep the backward compatibility

	"For example: DhcpEntryCommand=600, SavePasswordCommand=300, VmDataCommand=300", true);
	"For example: DhcpEntryCommand=600, SavePasswordCommand=300, VmDataCommand=300", false);

Introducing granular command timeouts global setting #9659

Are you sure you want to change the base?

Introducing granular command timeouts global setting #9659

Conversation

harikrishna-patnala commented Sep 9, 2024 • edited Loading

Description

Types of changes

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

harikrishna-patnala commented Sep 9, 2024

blueorangutan commented Sep 9, 2024

codecov bot commented Sep 9, 2024 • edited Loading

Codecov Report

blueorangutan commented Sep 9, 2024

harikrishna-patnala commented Sep 10, 2024

blueorangutan commented Sep 10, 2024

blueorangutan commented Sep 10, 2024

sureshanaparti left a comment

Choose a reason for hiding this comment

sureshanaparti commented Sep 10, 2024

blueorangutan commented Sep 10, 2024

blueorangutan commented Sep 11, 2024

harikrishna-patnala commented Sep 11, 2024

blueorangutan commented Sep 11, 2024

harikrishna-patnala commented Sep 11, 2024

blueorangutan commented Sep 11, 2024

blueorangutan commented Sep 11, 2024

sureshanaparti commented Sep 12, 2024

blueorangutan commented Sep 12, 2024

blueorangutan commented Sep 12, 2024

harikrishna-patnala commented Sep 13, 2024

blueorangutan commented Sep 13, 2024

blueorangutan commented Sep 13, 2024

harikrishna-patnala commented Sep 13, 2024

blueorangutan commented Sep 13, 2024

blueorangutan commented Sep 14, 2024

DaanHoogland Sep 16, 2024

Choose a reason for hiding this comment

harikrishna-patnala Sep 18, 2024

Choose a reason for hiding this comment

harikrishna-patnala commented Sep 18, 2024

blueorangutan commented Sep 18, 2024

blueorangutan commented Sep 18, 2024

harikrishna-patnala commented Sep 19, 2024

blueorangutan commented Sep 19, 2024

blueorangutan commented Sep 20, 2024

harikrishna-patnala commented Sep 20, 2024

blueorangutan commented Sep 20, 2024

blueorangutan commented Sep 20, 2024

harikrishna-patnala commented Sep 20, 2024

blueorangutan commented Sep 20, 2024

harikrishna-patnala commented Sep 9, 2024 •

edited

Loading

codecov bot commented Sep 9, 2024 •

edited

Loading