worker: Provide a sensible default for "critical load threshold"

Based on our experiences various issues with tests can appear if the
system load is exceeding certain sane levels, for example typing issues
such as mistyping, hanging keys, lost characters, but also various
timeouts and even network outages.

This commit introduces a new default value enabling the recently
introduced check for "critical load" on an openQA worker to prevent an
openQA worker from starting new jobs until the system load is again
below a threshold.

The value selection is of course based on empirical data gathered so far
and can and should be tweaked if we know more.

Related progress issue: https://progress.opensuse.org/issues/158125

etc/openqa/workers.ini

@@ -76,7 +76,10 @@
 # Specifies the threshold to consider the load on the machine critical. If the
 # average load (over the last 15 minutes) exceeds the specified value the worker
 # will not accept new jobs (until the load decreases again).
-#CRITICAL_LOAD_AVG_THRESHOLD = 10
+# The default value is 40 to prevent system overload based on experiences with
+# system stability so far.
+# Set to 0 to disable.
+#CRITICAL_LOAD_AVG_THRESHOLD = 40
 
 # The section ids are the instance of the workers.
 # The key/value pairs will appear in vars.json

lib/OpenQA/Worker/Settings.pm

@@ -65,6 +65,10 @@ sub new ($class, $instance_number = undef, $cli_options = {}) {
         }
     }
 
+    # Select sensible system CPU load15 threshold to prevent system overload
+    # based on experiences with system stability so far
+    $global_settings{CRITICAL_LOAD_AVG_THRESHOLD} //= 40;
+
     # set some environment variables
     # TODO: This should be sent to the scheduler to be included in the worker's table.
     if (defined $instance_number) {

t/24-worker-settings.t

@@ -20,6 +20,7 @@ my $settings = OpenQA::Worker::Settings->new;
 is_deeply(
     $settings->global_settings,
     {
+        CRITICAL_LOAD_AVG_THRESHOLD => 40,
         GLOBAL => 'setting',
         WORKER_HOSTNAME => '127.0.0.1',
         LOG_LEVEL => 'test',
@@ -82,6 +83,7 @@ subtest 'instance-specific settings' => sub {
     is_deeply(
         $settings1->global_settings,
         {
+            CRITICAL_LOAD_AVG_THRESHOLD => 40,
             GLOBAL => 'setting',
             WORKER_HOSTNAME => '127.0.0.1',
             WORKER_CLASS => 'qemu_i386,qemu_x86_64',
@@ -96,6 +98,7 @@ subtest 'instance-specific settings' => sub {
     is_deeply(
         $settings2->global_settings,
         {
+            CRITICAL_LOAD_AVG_THRESHOLD => 40,
             GLOBAL => 'setting',
             WORKER_HOSTNAME => '127.0.0.1',
             WORKER_CLASS => 'qemu_aarch64',