This document describes procedure how to setup kubernetes (k8s) cluster with CMS data-services using docker images. Here is a list of items we need:
- an account on CMS build (docker) node, e.g. cmsdev15
- an account on openstack.cern.ch
- an account on docker.com to upload your docker image
- a docker image(s) we want to deploy to k8s
- a kubernetes cluster where we'll deploy our image
We'll assume that you can get an account on CMS build node as well as on openstack.cern.ch
In order to access openstack nodes we need to create and upload rsa ssh key. You may create and upload your rsa key as following:
# create new key, here we give a key name as: cloud
ssh-keygen -t rsa -f cloud
# upload you key to openstack with name cloud
openstack keypair create --public-key ~/.ssh/cloud.pub cloud
For cmsweb operations we need to have a valid proxy on our backends. Originally, we used operator proxy and uploaded it to myproxy server. The cmsweb ProxySeed and ProxyRenew scripts were used.
Now, we can obtain CERN service account and apply for a Robot certificate. Once received and we obtain robotcert.pem/robotkey.pem files and we need to register them in LCG VOMS server. Then, we'll mount them in secrets volume in k8s cluster and use for acquiring proxies within k8s pods, see proxy.sh frontend file for example.
You can create your kubernetes cluster either by login to openstack.cern.ch and navigating to Container infra -> Cluster Templates and click on CREATE CLUSTER or you can login to lxplus-cloud.cern.ch (or lxplus7-cloud-testing) and create it via command line. We'll provide instruction here how to do the later. First, login to lxplus-cloud.cern.ch or lxplus7-cloud-testing.cern.ch. Then find out appropriate template
# list existing clusters
openstack coe cluster list
# list existing cluster templates
openstack coe cluster template list
# look-up details of concrete template
openstack coe cluster template show kubernetes
# here you need to create your ssh keypair (named cloud here)
# and upload it to openstack.cern.ch
openstack coe cluster create vkcluster --keypair cloud --cluster-template kubernetes-preview
# at later time the cluster can be deleted as following
openstack coe cluster delete vkcluster
# to deal with different project-name namespace we need to use the following commands
# here "CMS Webtools Mig" is our project-name namespace
# cmswebk8s is a name of a cluster we create and delete
openstack --os-project-name "CMS Webtools Mig" coe cluster list
# example of create command
openstack --os-project-name "CMS Webtools Mig" coe cluster create cmswebk8s --keypair cloud --cluster-template kubernetes-preview --flavor m2.2xlarge --node-count 2
# another example to allow every node in a cluster to have
# host certificates and krb5
openstack --os-project-name "CMS Webtools Mig" coe cluster create k8s --keypair cloud --cluster-template kubernetes-preview --flavor m2.2xlarge --labels cern_enabled=True
openstack --os-project-name "CMS Webtools Mig" coe cluster delete cmswebk8s
Here, we first list existing clusters, then existing template kubernetes and finally created
a custom cluster named vkcluster
(change to your name). Please note, that you must
name your cluster with lower-case letters and no special symbols. Then, we can verify
that our cluster is up and running by using the following command:
openstack coe cluster show vkcluster
and in order to access it and plays with kubernetes commands we should setup the environment
cd your_dir
$(openstack coe cluster config vkcluster)
This will create few files: ca.pem
, cert.pem
, key.pem
and config
. At this time
the environment will be set and we can start using kubectl
command. But if you'll login
next time you'll need to setup your environment by simply doing
export KUBECONFIG=/path/config
where /path
is your area where you created aforementioned files.
Please refer to [2] for kubernetes tutorial which describe the terminology.
Here we're going to use kubectl
command to create pod and our service.
Let's start with inspecing existing node(s):
kubectl get node
NAME STATUS AGE
vkcluster-dzegay3ktjak-minion-0 Ready 1d
Now, we can create and deploy our docker image as following (please substitute USERNAME and port number accoringly):
# create new app from docker image to be accessible on given port
kubectl run kubernetes-das2go --image=USERNAME/das2go --port=8212
kubectl get pods
kubectl get deployments
export POD_NAME=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
kubectl logs $POD_NAME
# connect to my pod
kubectl exec -ti $POD_NAME bash
# now I should have a bash shell prompt on my pod
exit # exit from the shell/pod
# if we want to delete our deployment we'll do
kubectl delete deployment kubernetes-das2go
We already listed command how you can login into your pod and inspect your app within it. It's time to expose it to be visible outside of internal cluster.
# expose our services to outside world
kubectl expose deployment/kubernetes-das2go --type="NodePort" --port 8212
service "kubernetes-das2go" exposed
kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.254.0.1 <none> 443/TCP 3d
kubernetes-das2go 10.254.138.221 <nodes> 8212:30695/TCP 7s
# check our service information
kubectl describe services/kubernetes-das2go
Name: kubernetes-das2go
Namespace: default
Labels: run=kubernetes-das2go
Selector: run=kubernetes-das2go
Type: NodePort
IP: 10.254.138.221
Port: <unset> 8443/TCP
NodePort: <unset> 30695/TCP
Endpoints: 10.100.39.6:8212
Session Affinity: None
No events.
# if we want to delete our service we'll do
kubectl delete services -l run=kubernetes-das2go
Above we shown that our service is up and running and has open port 30695
.
This port and IP addresses above will change in your setup. In order to
access our app we need to know an external IP and PORT numbers. Here how
we can find them:
# get external port number from service template
export NODE_PORT=$(kubectl get services/kubernetes-das2go -o go-template='{{(index .spec.ports 0).nodePort}}')
# or inspect service information
# obtain public service IP which is visible at CERN network
openstack coe cluster show vkcluster | grep node_addresses
# now I can access my service as
host=`openstack coe cluster show vkcluster | grep node_addresses | awk '{print $4}' | sed -e "s,\[u',,g" -e "s,'\],,g"`
curl http://$host:$NODE_PORT/das/
We are done.
- http://clouddocs.web.cern.ch/clouddocs/containers/quickstart.html
- https://kubernetes.io/docs/tutorials/kubernetes-basics/
- https://kubernetes.io/docs/reference/kubectl/docker-cli-to-kubectl/
- http://blog.kubernetes.io/2015/10/some-things-you-didnt-know-about-kubectl_28.html
- https://docs.openstack.org/magnum/latest/contributor/quickstart.html