Guide to prepare Openshift cluster to run testsuite

Requirements

Software	Source
OpenShift cluster*	https://docs.openshift.com/container-platform/4.14/installing/index.html
OpenShift client	https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/
Kuadrant operator	Available in Openshift OperatorHub
Sail operator (3.0-dp1)	Available in Openshift OperatorHub
cert-manager (min. v1.12.1)	Available in Openshift OperatorHub
Gateway API CRD	https://github.com/kubernetes-sigs/gateway-api/releases/
testsuite tools	https://github.com/3scale-qe/tools

* Version 4.14+ with an admin account

Optional

Software	Source
AWS Route53 account	https://aws.amazon.com/route53/

Step 0 Log in to the Openshift cluster and create namespaces

Use oc command to log in to target the Openshift cluster.
Create required namespaces for the deployment:
- istio-system this will hold Sail operator and Istio objects
- kuadrant-system this will hold Kuadrant operator and Kuadrant CR object
- kuadrant this will hold test objects
- kuadrant2 this will hold additional test objects
- tools this will hold tools needed for testing

oc new-project "istio-system" &&\
oc new-project "kuadrant-system" &&\
oc new-project "kuadrant" &&\
oc new-project "kuadrant2" &&\
oc new-project "tools"

Step 1 Istio deployment: Sail operator

Upload OperatorGroup.

oc create -n "istio-system" -f - <<EOF
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: istio-operator-group
spec:
  upgradeStrategy: Default
EOF

Upload Subscription object which will install Sail operator.

oc create -n "istio-system" -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sailoperator
spec:
  channel: 3.0-dp1
  installPlanApproval: Automatic
  name: sailoperator
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: sailoperator.v3.0.0-dp1
EOF

Wait for installation to finish.

oc wait -n "istio-system" subscriptions/sailoperator --for=jsonpath='{.status.installPlanRef.name}' && ISTIO_PLAN=$(oc get -n "istio-system" subscriptions/sailoperator -o "jsonpath={.status.installPlanRef.name}") && oc wait -n "istio-system" InstallPlan/$ISTIO_PLAN --for=condition=Installed=True

Create new Istio CR with name "defualt".

oc create -n "istio-system" -f - <<EOF
apiVersion: operator.istio.io/v1alpha1
kind: Istio
metadata:
  name: default
spec:
  updateStrategy:
    inactiveRevisionDeletionGracePeriodSeconds: 30
    type: InPlace
  namespace: istio-system
  version: v1.21.0
  values:
     pilot:
      autoscaleMin: 2
     global:
      hub: gcr.io/istio-release
EOF

Wait until Istio CR is healthy.

oc wait -n "istio-system" istio/default --for=condition=Ready=True

Install additional CRD for Gateway API objects. For more information visit: https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api

oc apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/standard-install.yaml

Step 2 Kuadrant deployment: OperatorGroup and Kuadrant operator

In this step, the Kuadrant operator will be installed.

Upload OperatorGroup.

oc create -n "kuadrant-system" -f - <<EOF
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: kuadrant-operator-group
spec:
  upgradeStrategy: Default
EOF

Create a CatalogSource to control deployed Kuadrant version by changing the image url. Available tags: https://quay.io/repository/kuadrant/kuadrant-operator-catalog?tab=tags

oc create -n "openshift-marketplace" -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: kuadrant-upstream
spec:
  image: 'quay.io/kuadrant/kuadrant-operator-catalog:latest'
  sourceType: grpc
EOF

Wait for the CatalogSource to be ready.

oc wait -n "openshift-marketplace" Catalogsource/kuadrant-upstream --for=jsonpath='{.status.connectionState.lastObservedState}'=READY

Upload the Subscription object that will install the Kuadrant operator. Choose install channel depending on what version of catalog source you chosen before. preview for nightly builds and stable for release builds.

oc create -n "kuadrant-system" -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: kuadrant-operator
spec:
  channel: preview
  installPlanApproval: Automatic
  name: kuadrant-operator
  source: kuadrant-upstream
  sourceNamespace: openshift-marketplace
EOF

Wait for the installation to be complete.

oc wait -n "kuadrant-system" subscriptions/kuadrant-operator --for=jsonpath='{.status.installPlanRef.name}' && KUADRANT_PLAN=$(oc get -n "kuadrant-system" subscriptions/kuadrant-operator -o "jsonpath={.status.installPlanRef.name}") && oc wait -n "kuadrant-system" InstallPlan/$KUADRANT_PLAN --for=condition=Installed=True --timeout=180s

Deploy Kuadrant instance.

oc create -n "kuadrant-system" -f - <<EOF
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant-sample
spec: {}
EOF

Wait until Kuadrant is ready.

oc wait -n "kuadrant-system" kuadrant/kuadrant-sample --for=condition=Ready=True

Step 3 Testsuite tools

In this step, the supporting software tools will be installed.

Deploy the tools.

oc apply -k https://github.com/3scale-qe/tools.git/overlays/kuadrant --namespace tools

Enable tracing for Istio

Configure tracing for istio/default to use Jager.

oc patch -n istio-system istio/default --type=merge -p '{"spec": {"values": {"meshConfig":{"defaultConfig":{"tracing":{}},"enableTracing":true},"global":{"proxy":{"logLevel": "info"}}}}}' &&\
oc patch -n istio-system istio/default --type=json -p '[{"op": "add", "path": "/spec/values/meshConfig/extensionProviders/-", "value": {"name": "jaeger-otlp","opentelemetry":{"service":"jaeger-collector.tools.svc.cluster.local","port":4317}}}]'

Enable tracing.

oc create -n "istio-system" -f - <<EOF
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
spec:
  tracing:
  - providers:
    - name: jaeger-otlp
    randomSamplingPercentage: 100
EOF

Step 4 Cert-manager setup (Optional)

Required for running DNSPolicy and TLSPolicy tests. More info for setting up the Letsecrypt Issuer: https://cert-manager.io/docs/configuration/acme/dns01/route53/

You can install either the community cert-manager (shown in this guide, cluster-scoped) or Red Hat cert-manager operator (it is namespace-scoped so both OperatorGroup and Subscription resources need to be created (or use Operator Hub UI). Install it into "kuadrant" namespace).

Create Subscription that will install cert-manager.

oc create -n openshift-operators -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cert-manager
spec:
  channel: stable
  installPlanApproval: Automatic
  name: cert-manager
  source: community-operators
  sourceNamespace: openshift-marketplace
EOF

Wait until installation finishes:

oc wait -n "openshift-operators" subscriptions/cert-manager --for=jsonpath='{.status.installPlanRef.name}' && INSTALL_PLAN=$(oc get -n "openshift-operators" subscriptions/cert-manager -o "jsonpath={.status.installPlanRef.name}") && oc wait -n "openshift-operators" InstallPlan/$INSTALL_PLAN --for=condition=Installed=True

Create self-signed ClusterIssuer.

oc create -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
EOF

Create secret for DNS-01 verification for Letsencrypt issuer. You need to supply your secret key.

oc create -n "kuadrant" -f - <<EOF
kind: Secret
apiVersion: v1
metadata:
  name: aws-secret
data:
  awsSecretAccessKey: <SECRET>
type: Opaque
EOF

Fill required fields and create Letsecrypt Issuer.

oc create -n "kuadrant" -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging-issuer
spec:
  acme:
    email: <ACCOUNT EMAIL>
    preferredChain: ISRG Root X1
    privateKeySecretRef:
      name: letsencrypt-private-key
    server: 'https://acme-staging-v02.api.letsencrypt.org/directory'
    solvers:
      - dns01:
          route53:
            accessKeyID: <ID>
            hostedZoneID: <ID>
            region: <REGION>
            secretAccessKeySecretRef:
              key: awsSecretAccessKey
              name: aws-secret
EOF

Check the Issuer is ready.

oc wait -n kuadrant Issuer/letsencrypt-staging-issuer --for=condition=Ready

Restart Kuadrant Operator

oc delete pod $(oc get pod -n kuadrant-system | grep kuadrant-operator | awk '{print $1}') -n kuadrant-system

Step 5 DNS provider secret (Optional)

Required for running DNSPolicy and TLSPolicy tests.

Create AWS key with "base_domain" annotation. You need to supply your secret key, id, and region.

oc create -n "kuadrant" -f - <<EOF
kind: Secret
apiVersion: v1
metadata:
  name: aws-credentials
  annotations:
    base_domain: example.com
stringData:
  AWS_ACCESS_KEY_ID: <ID>
  AWS_REGION: <REGION>
  AWS_SECRET_ACCESS_KEY: <KEY>
type: kuadrant.io/aws
EOF

Step 6 Testsuite run

Finally run tests with make in the root of testsuite repository:

Run make test to run all Kuadrant tests.
Run make authorino-standalone to run Authorino tests without Kuadrant

Provide feedback

Saved searches

Use saved searches to filter your results more quickly