Canary deployments - REST, TW, Publisher

[mapellidario]

introduction

We have always dreamed about processing a small portion of production crab tasks with a new crab version.

We do not have a general solution, each crab system will need to be adapted in a very specific way.

REST

This requires no change in the code, but requires some experience with k8s deployments. The idea is:

• create a new deployment in dmwm/CMSKubernetes/helm/crabserver/templates, call it crabserver-canary.
  • change only metadata.name: crabserver-canary, do not change for example metadata.labels.app: crabserver nor spec.selector.matchLabels.app: crabserver
• create a new values file dmwm/CMSKubernetes/helm/crabserver/values-canary.yaml
• change deploy.sh to accept canary as an argument, you will need to edit the cluster_map.
  • otherwise simply use helm template crabserver . -f values.yaml -f values-${1}.yaml | kubectl -n crab apply -f -, but beware to be connected with the proper context (username+cluster)!

TaskWorker

This requires some change in the code, that have been outlined in https://github.com/dmwm/CRABServer/wiki/TaskWorker-Canary-Deployment

we decided that we run one TW per virtual machine, so that the hostname is enough to identify which TW process/container we are referring to.

Publisher

This is not defined yet, but we have some ideas. The simplest one is

• run a publisher on every VM where we run TW
• make sure that a single publisher only select jobs that have been submitted by tasks processed by the TW running on the same VM
  • need to add tm_twname to filetransfersdb table or pick it up from task table using a join on taskname. In any case need to change the SQL which picks the files to work on.

[belforte]

I believe that we agreed that @novicecpp will do the K8s part together with @aspiringmind-code

I am still un-decided if it is better to call it crabserver-canary or crabserver-qa

[belforte]

NOTE: it will be important to be able to tell (easily) the canary pod from the others in our monitoring, e.g. quickly tell where HTTP errors come from

[novicecpp]

Look promising.
https://monit-grafana.cern.ch/goto/3JXSKT3Ig?orgId=11

EDIT: new image to contains pod name.

[belforte]

better split off TW+Publisher to a different issue #8678

[novicecpp]

To-do:

• Waiting Merge crab rest, log pipeline, and serviceaccount manifests together in single helm chart #8677 to close
• Deploy CRABServer helm: deploy all crab services in single helm chart CMSKubernetes#1545 on production
• Test it on production (same image tag) to verify dashboard.

[novicecpp]

Sorry. I need to keep this open because it does not deploy on production yet.

test (at least test12) and preprod are now use new helm chart.

[belforte]

@aspiringmind-code will do it :-)

[novicecpp]

Sorry, wrong issue.

This one I tested myself on my test12 and preprod, and it works.
But you may need to modify dashboard a bit to make it more easier to see metrics between crabserver and crabserver-canary.

[belforte]

I knew that it works, but AFAIK it is not deployed in production, not used, not monitored. We have not even deployed latest CRABServer tag there. Plenty of useful work to do !