From f9f6b21460b676518424c23f5226cd05cd0f3042 Mon Sep 17 00:00:00 2001 From: Marius Kittler Date: Tue, 23 Jan 2024 16:35:13 +0100 Subject: [PATCH 1/5] Mention specific test scenarios that can be used to verify the MM setup * Mention specific names of wicked tests so one knows what to search for * Mention the `ping_server`/`ping_client` scenario which has recently been introduced to test the MM setup via openQA itself * See https://progress.opensuse.org/issues/135914 --- docs/Networking.asciidoc | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/Networking.asciidoc b/docs/Networking.asciidoc index 091980ae71f..1c26c5e64ec 100644 --- a/docs/Networking.asciidoc +++ b/docs/Networking.asciidoc @@ -191,8 +191,9 @@ systemctl enable openqa-worker@{1..3} === Verify the setup Simply run a MM test scenario. For openSUSE, you can find many relevant tests on https://openqa.opensuse.org[o3], e.g. look for networking-related tests like -wicked-tests. To test GRE tunnels, you may want to change the jobs worker -classes so the different jobs are executed on different workers. +`ping_server`/`ping_client` or `wicked_basic_ref`/`wicked_basic_sut`. To test +GRE tunnels, you may want to change the jobs worker classes so the different +jobs are executed on different workers. So you could call `openqa-clone-job` like this: ``` From 81710be1336fc8effa10fe7a1accaf5cba34a69f Mon Sep 17 00:00:00 2001 From: Marius Kittler Date: Tue, 23 Jan 2024 16:37:47 +0100 Subject: [PATCH 2/5] Improve documentation for running MM jobs accross multiple hosts * Improve how sentences are divided into paragraphs * Use the recently introduced `:`-syntax to specify test-specific worker classes instead of using `--export-command` * See https://progress.opensuse.org/issues/135914 --- docs/Networking.asciidoc | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/docs/Networking.asciidoc b/docs/Networking.asciidoc index 1c26c5e64ec..abb21a242a3 100644 --- a/docs/Networking.asciidoc +++ b/docs/Networking.asciidoc @@ -191,23 +191,21 @@ systemctl enable openqa-worker@{1..3} === Verify the setup Simply run a MM test scenario. For openSUSE, you can find many relevant tests on https://openqa.opensuse.org[o3], e.g. look for networking-related tests like -`ping_server`/`ping_client` or `wicked_basic_ref`/`wicked_basic_sut`. To test -GRE tunnels, you may want to change the jobs worker classes so the different -jobs are executed on different workers. +`ping_server`/`ping_client` or `wicked_basic_ref`/`wicked_basic_sut`. -So you could call `openqa-clone-job` like this: +To test GRE tunnels, you may want to change the jobs worker classes so the +different jobs are executed on different workers. So you could call +`openqa-clone-job` like this: ``` openqa-clone-job \ - --skip-download --skip-chained-deps \ # assuming assets are present - --max-depth 0 \ # clone the entire parallel cluster - --export-command \ # only print the API call - https://openqa.opensuse.org/tests/250309 # arbitrary job in cluster to clone - _GROUP=0 BUILD+=test-mm-setup # avoid interfering with production jobs + --skip-download --skip-chained-deps \ # assuming assets are present + --max-depth 0 \ # clone the entire parallel cluster + https://openqa.opensuse.org/tests/3886213 \ # arbitrary job in cluster to clone + _GROUP=0 BUILD+=test-mm-setup \ # avoid interfering with production jobs + WORKER_CLASS:wicked_basic_ref+=,worker_foo \ # ensure one job runs on `worker_foo` + WORKER_CLASS:wicked_basic_sut+=,worker_bar # ensure other job runs on `worker_bar` ``` -It will print an `openqa-cli` call. You can modify it to change the worker -classes of the jobs individually and then invoke it. - Also be sure to reboot the worker host to make sure the setup is actually persistent. From 701ce46a82342de60fe5eb61336d28ed6b921eb5 Mon Sep 17 00:00:00 2001 From: Marius Kittler Date: Tue, 23 Jan 2024 16:45:50 +0100 Subject: [PATCH 3/5] Update MTU-related documentation in MM-setup documentation * Update the concrete MTU to what we now use in the openSUSE test distribution * Mention how to do ping tests with different MTU sizes * See https://progress.opensuse.org/issues/135914 --- docs/Networking.asciidoc | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/Networking.asciidoc b/docs/Networking.asciidoc index abb21a242a3..4c2ea94348b 100644 --- a/docs/Networking.asciidoc +++ b/docs/Networking.asciidoc @@ -231,7 +231,7 @@ Within the VM configure the network *like* this (you may need to adjust concrete IP addresses, subnets and interface names): ``` -ip link set dev eth0 up mtu 1458 +ip link set dev eth0 up mtu 1380 ip a add dev eth0 10.0.2.15/24 ip r add default via 10.0.2.2 echo 'nameserver 8.8.8.8' > /etc/resolv.conf @@ -242,7 +242,10 @@ for MM tests and should be below the MTU set on the Open vSwitch bridge device (e.g. via `os-autoinst-setup-multi-machine`). After this it should be possible to reach other hosts. You may also launch a 2nd -VM to see whether the VMs can talk to each other. +VM to see whether the VMs can talk to each other. You may conduct ping tests +similar to the `ping_client` test mentioned in the previous section (see the +https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/cc3a5b32527c4c8bb810c8bce9b1449a891ef74b/lib/utils.pm#L2901[utility function in openSUSE tests] +for details). === Debugging Open vSwitch Configuration Boot sequence with wicked (version 0.6.23 and newer): From 5cb0385b85df765d850c873f2c8708c676d2dc22 Mon Sep 17 00:00:00 2001 From: Marius Kittler Date: Tue, 23 Jan 2024 16:54:37 +0100 Subject: [PATCH 4/5] Document command to check whether IP forwarding is enabled * This is something to check when the MM setup doesn't work, see e.g. https://progress.opensuse.org/issues/134282#note-105 * See https://progress.opensuse.org/issues/135914 --- docs/Networking.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Networking.asciidoc b/docs/Networking.asciidoc index 4c2ea94348b..fd10be7f64d 100644 --- a/docs/Networking.asciidoc +++ b/docs/Networking.asciidoc @@ -261,6 +261,7 @@ The configuration and operation can be checked with the following commands: [source,sh] ---- +cat /proc/sys/net/ipv4/conf/{br1,eth0}/forwarding # check whether IP forwarding is enabled ovs-vsctl show # shows the bridge br1, the tap devices are assigned to it ovs-ofctl dump-flows br1 # shows the rules installed by os-autoinst-openvswitch in table=0 ovs-dpctl show # show basic info on all datapaths From 2264b0323b01e4e22c955a7b6b752508ef15c46f Mon Sep 17 00:00:00 2001 From: Marius Kittler Date: Tue, 23 Jan 2024 17:06:56 +0100 Subject: [PATCH 5/5] Document debugging problematic connections in the GRE network * As it was done in https://progress.opensuse.org/issues/152389 * See https://progress.opensuse.org/issues/135914 --- docs/Networking.asciidoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/Networking.asciidoc b/docs/Networking.asciidoc index fd10be7f64d..87fd09c1874 100644 --- a/docs/Networking.asciidoc +++ b/docs/Networking.asciidoc @@ -245,7 +245,9 @@ After this it should be possible to reach other hosts. You may also launch a 2nd VM to see whether the VMs can talk to each other. You may conduct ping tests similar to the `ping_client` test mentioned in the previous section (see the https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/cc3a5b32527c4c8bb810c8bce9b1449a891ef74b/lib/utils.pm#L2901[utility function in openSUSE tests] -for details). +for details). When running ping you can add/remove machines to/from the GRE +network to bisect problematic hosts/connections (via `ovs-vsctl add-port …` and +`ovs-vsctl del-port …`). === Debugging Open vSwitch Configuration Boot sequence with wicked (version 0.6.23 and newer):