Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI integration of networkservicemesh integration tests (NSMNSE-79) #23

Open
wants to merge 64 commits into
base: vl3_latest
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 62 commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
25366f8
Initial commit
Jan 27, 2021
5159511
Add stage for installing docker client to jenkinsfile
Jan 27, 2021
f9c29c9
Add stage for installing yq in jenkinsfile
Jan 27, 2021
b5756fa
Add stage for installing kubectl in jenkinsfile
Jan 27, 2021
7656f1c
escape dollar sign
Jan 27, 2021
3215b2d
Add stage to install helm to jenkinsfile
Jan 27, 2021
8377e22
Add environment vars to jenkinsfile
Jan 27, 2021
2f1e4f9
Add stage to install kind to jenkinsfile
Jan 27, 2021
8e3fa80
Add stage to install cloudtest to jenkinsfile
Jan 27, 2021
aeef96b
Add stage to run tests in jenkinsfile
Jan 27, 2021
e6d59f9
Add docker host networking to jenkinsfile
Jan 27, 2021
51afdd1
Run single cluster tests in jenkins
Jan 27, 2021
93ccd37
Run single cluster tests in jenkins
Jan 27, 2021
8239b95
Increase cloudtest timeout to 2 hours
Jan 28, 2021
9ca800f
Add 'wcm' label to jenkinsfile so it runs on the right nodes
Jan 28, 2021
38cd1d4
modify the way kind is installed in jenkinsfile
Jan 28, 2021
f2520e4
reduce stats prints frequency and cleanup jenkinsfile
Jan 28, 2021
b8edfc1
create two kind clusters for interdomain tests
Feb 1, 2021
0365ef6
try again to create two kind clusters for interdomain tests
Feb 1, 2021
2f52c5b
move all config into cloudtest config file
Feb 1, 2021
a355cf9
use separate cloudtest config files for now to make it easier to trig…
Feb 1, 2021
4ccfbae
add interdomain tests to the jenkinsfile in a separate stage
Feb 1, 2021
95553fa
add cluster selector to interdomain tests to see if that helps them run
Feb 1, 2021
aedaf82
try adding a second kind cluster selector to interdomain tests to see…
Feb 1, 2021
ddfbba0
fix prepare section of cloudtest config
Feb 1, 2021
04d69f4
use hardcoded cluster names to see if it will run
Feb 1, 2021
d7007c4
change instances to 1 to see if that helps
Feb 1, 2021
f2eba70
try to fix path to kubeconfig
Feb 1, 2021
01ffb5b
pass kubeconfig locations directly into go test
Feb 1, 2021
cbbe6b9
catch errors in interdomain tests
Feb 1, 2021
b92b097
change 'steps' to 'step' in jenkinsfile
Feb 1, 2021
a5906a0
use try/catch in jenkinsfile instead of catchError inside step
Feb 1, 2021
e155706
add another interdomain test to the jenkinsfile
Feb 1, 2021
d0ca32f
try to run interdomain tests using cloudtest
Feb 2, 2021
a5e53c3
modify cluster-selector in cloudtest interdomain test config
Feb 2, 2021
9ebeca5
try two different kind providers
Feb 2, 2021
42e0393
fix yaml file
Feb 2, 2021
4a842fa
try to save test results as an artifact
Feb 2, 2021
02a03e9
try to save test results as an artifact
Feb 2, 2021
5addda7
try to save test results as an artifact
Feb 2, 2021
e74f800
try to save test results as an artifact
Feb 2, 2021
8140068
try to save test results as an artifact
Feb 2, 2021
1978eda
add single cluster tests to junit reporting and save report as an art…
Feb 2, 2021
68931b1
add cloudtest logs to artifacts in jenkinsfile
Feb 2, 2021
59817d6
add everything under .tests/cloud_test dir to artifacts in jenkinsfile
Feb 2, 2021
b8fded3
add interdomain tests back in
Feb 2, 2021
c188e63
try to save test results as an artifact
Feb 3, 2021
10f96b8
clean up jenkins file
Feb 3, 2021
1c68a87
archive artifacts for more often
Feb 3, 2021
815833b
catch test errors so that future test stages can run in jenkinsfile
Feb 3, 2021
bcac0b7
store cloudtest artificats for each group of tests in a separate dire…
Feb 3, 2021
6912e0f
a
Feb 3, 2021
58debbe
add back in interdomainl
Feb 3, 2021
cd31d11
find files recursively to archive
Feb 3, 2021
8fb9d7a
a
Feb 4, 2021
8064ba6
a
Feb 4, 2021
cbb7aa4
a
Feb 4, 2021
a15c242
remove unused files
Feb 4, 2021
23b4804
remove more unused files
Feb 4, 2021
3c7e2ac
use helm 3 instead of helm 2
Feb 4, 2021
6d85130
use helm 3 instead of helm 2
Feb 4, 2021
a3c03fb
undo changes to helm-init-wrapper.sh
Feb 4, 2021
34146a0
add back deleted .cloudtest/execution folder and files to minimize ch…
Feb 5, 2021
1f6abff
revert changes to go.mod and go.sum
Feb 5, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions .cloudtest-bench.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
version: 1.0
root: "./.tests/cloud_test/bench/"
timeout: 7200 # 2 hour total total timeout
shuffle-enabled: false
statistics:
enabled: true
interval: 30 # 30 seconds for statistics
providers:
- name: "kind" # A name of our provider
kind: "shell" # A type of provider, shell indicate we need to specify all operations with shell commands.
instances: 1 # We need to have 1 instance of cloud.
node-count: 3 # With 3 nodes, CloudTest will validate if cloud has 3 nodes alive.
retry: 1 # A retry count if cloud is failed to start tool will try again.
enabled: true # Indicate Provider is active with config by default, if set to false, values could be overriden using command line arguments.
timeout: 300 # 5 minutes to start cluster
stop-delay: 10
env:
- KIND_CLUSTER_NAME=cloudtest-kind-$(rands10) # Generate a uniq cluster name
- CONFIG_LOCATION=$(tempdir)/config # Put Kubernetes configuration file here, $(tempdir) a executable variable name pointing to temporary folder.
scripts:
start: make kind-start
config: make kind-export-kubeconfig # Use make file to generate kind config and put it into a file specified with CONFIG_LOCATION environment variable.
stop: make kind-stop
prepare: |
make k8s-load-images
make spire-install
make k8s-config
executions:
- name: "Benchmark tests"
env:
- ARTIFACTS_ARCHIVE=true
source:
tags:
- bench
root: ./test/integration
timeout: 600
cluster-count: 1
cluster-env:
- KUBECONFIG
on-fail: |
make k8s-reset
reporting:
junit-report: "results/junit-benchmark-tests.xml"
49 changes: 49 additions & 0 deletions .cloudtest-interdomain.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
version: 1.0
root: "./.tests/cloud_test/interdomain/"
timeout: 7200 # 2 hour total total timeout
shuffle-enabled: false
statistics:
enabled: true
interval: 30 # 30 seconds for statistics
providers:
- name: "kind" # A name of our provider
kind: "shell" # A type of provider, shell indicate we need to specify all operations with shell commands.
instances: 2
node-count: 3 # With 3 nodes, CloudTest will validate if cloud has 3 nodes alive.
retry: 1 # A retry count if cloud is failed to start tool will try again.
enabled: true # Indicate Provider is active with config by default, if set to false, values could be overriden using command line arguments.
timeout: 300 # 5 minutes to start cluster
stop-delay: 10
env:
- KIND_CLUSTER_NAME=cloudtest-kind-$(rands10) # Generate a uniq cluster name
- CONFIG_LOCATION=$(tempdir)/config # Put Kubernetes configuration file here, $(tempdir) a executable variable name pointing to temporary folder.
scripts:
start: make kind-start
config: make kind-export-kubeconfig # Use make file to generate kind config and put it into a file specified with CONFIG_LOCATION environment variable.
stop: make kind-stop
prepare: |
make k8s-load-images
make spire-install
make k8s-config
executions:
- name: "Interdomain tests"
env:
- ARTIFACTS_ARCHIVE=true
- USE_JAEGER_SERVICE=true
source:
tags:
- interdomain
root: ./test/integration
timeout: 600
cluster-count: 2
cluster-selector:
- kind
- kind
cluster-env:
- KUBECONFIG_CLUSTER_1
- KUBECONFIG_CLUSTER_2
on-fail: |
make k8s-reset
reporting:
junit-report: "results/junit-interdomain-tests.xml"
46 changes: 46 additions & 0 deletions .cloudtest-single-cluster.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
version: 1.0
root: "./.tests/cloud_test/single-cluster/"
timeout: 7200 # 2 hour total total timeout
shuffle-enabled: false
statistics:
enabled: true
interval: 30 # 30 seconds for statistics
providers:
- name: "kind" # A name of our provider
kind: "shell" # A type of provider, shell indicate we need to specify all operations with shell commands.
instances: 1 # We need to have 1 instance of cloud.
node-count: 3 # With 3 nodes, CloudTest will validate if cloud has 3 nodes alive.
retry: 1 # A retry count if cloud is failed to start tool will try again.
enabled: true # Indicate Provider is active with config by default, if set to false, values could be overriden using command line arguments.
timeout: 300 # 5 minutes to start cluster
stop-delay: 10
env:
- KIND_CLUSTER_NAME=cloudtest-kind-$(rands10) # Generate a uniq cluster name
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does the injection of the $(rands10) actually work? Does cloudtest process the yaml file content in a way that a bash shell would exec that?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see that a makefile target is wrapping usage of the env var with $() exec... interesting

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cloudtest does do extra processing.
https://github.com/networkservicemesh/cloudtest/blob/master/docs/README.md#environment-variables-processing

you get environment variables and a handful of other useful commands, like rands10. I'm not sure if this is executed by bash or not, they may have just borrowed the syntax.

- CONFIG_LOCATION=$(tempdir)/config # Put Kubernetes configuration file here, $(tempdir) a executable variable name pointing to temporary folder.
scripts:
start: make kind-start
config: make kind-export-kubeconfig # Use make file to generate kind config and put it into a file specified with CONFIG_LOCATION environment variable.
stop: make kind-stop
prepare: |
make k8s-load-images
make spire-install
make k8s-config
executions:
- name: "Single cluster tests"
env:
- ARTIFACTS_ARCHIVE=true
source:
tags:
- basic
- recover
- usecase
root: ./test/integration
timeout: 300
cluster-count: 1
cluster-env:
- KUBECONFIG
on-fail: |
make k8s-reset
reporting:
junit-report: "results/junit-single-cluster-tests.xml"
36 changes: 0 additions & 36 deletions .cloudtest.yaml

This file was deleted.

17 changes: 0 additions & 17 deletions .cloudtest/executions/benchmark-tests.yaml

This file was deleted.

16 changes: 0 additions & 16 deletions .cloudtest/executions/example-crossconnect-monitor.yaml

This file was deleted.

21 changes: 0 additions & 21 deletions .cloudtest/executions/example-helm-interdomain.yaml

This file was deleted.

18 changes: 0 additions & 18 deletions .cloudtest/executions/example-icmp-kernel-forwarder-wireguard.yaml

This file was deleted.

17 changes: 0 additions & 17 deletions .cloudtest/executions/example-icmp-kernel-forwarder.yaml

This file was deleted.

16 changes: 0 additions & 16 deletions .cloudtest/executions/example-icmp.yaml

This file was deleted.

16 changes: 0 additions & 16 deletions .cloudtest/executions/example-vpn.yaml

This file was deleted.

16 changes: 0 additions & 16 deletions .cloudtest/executions/example-vpp-icmp.yaml

This file was deleted.

Loading