The CMS topic collector is to be deployed just like any other CERN WEB service.
The CMS Topic Collector is just a bunch of state-less cgi scripts, so you can deploy them on any CGI enabled web server with SSO support. For the CMS Official Instance you can simply rely on the CERN web service support.
First of all go to:
https://webservices.web.cern.ch/webservices/
create a new AFS based web site. We reccomend using your AFS workspace for
doing so, make also sure you enable CGI support. In the case of the Official
Instace, we use the cmsbuild account and
/afs/cern.ch/work/c/cmsbuild/www/cmsgit.
Assuming your site is in:
WORKDIR=/afs/cern.ch/work/c/cmsbuild
CMSGIT_ROOT=$WORKDIR/www/cmsgit
you'll need to make sure the web server can read those files:
fs sa -dir $WORKDIR -acl webserver:afs l
fs sa -dir $WORKDIR/www -acl webserver:afs l
fs sa -dir $CMSGIT_ROOT -acl webserver:afs rl
fs sa -dir $CMSGIT_ROOT -acl system:anyuser none
you'll also want to create a parallel directory which contains data, but which is not at all exposed to the web:
mkdir -p $WORKDIR/cmsgit-data
fs sa -dir $WORKDIR/cmsgit-data -acl webserver:afs rl
fs sa -dir $WORKDIR/cmsgit-data -acl system:anyuser none
notice you will need to make sure the webserver:afs user can list (but not
read!) the parents of the directories you have picked up.
You then need to enable SSO, i.e. at CERN you'll need to do:
cat << \EOF > $CMSGIT_ROOT/.htaccess
ShibRequireAll On
ShibRequireSession On
ShibExportAssertion On
SSLRequireSSL # The modules only work using HTTPS
AuthType Shibboleth
Require adfs-group "User admin group ZH" "cms-zh"
EOF
This will give you a working CGI area, accessible only by CMS users. To test that:
cat << \EOF > $CMSGIT_ROOT/hello
#!/usr/bin/env python
from os import environ
print "Status: 200 OK"
print
print "Hello %s!" % environ["ADFS_LOGIN"]
EOF
chmod +x $CMSGIT_ROOT/hello
and then check the web page works correctly. In case of the Official Instance simply go to:
https://cern.ch/cmsgit/hello
If everything works correctly it should ask you to login and show:
Hello <your-sso-login>!
Remove $CMSGIT_ROOT/hello, and go to next session.
Now it's time to populate the web area with the actual sources of the topic collector.
cd $CMSGIT_ROOT
git clone --bare [email protected]:www/cgi-bin/git-collector/.git .git
git init
git read-tree -mu HEAD
from now on you can update them by simply doing git pull origin master.
Given the Topic Collector is only a thin layer on top of the GitHub API, we need to get github API secrets to make sure we can access the repository of our choice.
If you are deploying the official CMS Topic Collector, you do not need to do any of the following and you can skip to the next section.
First of all register a new
application and note down the
client ID and client secret we will refer to them as CLIENT_ID and
CLIENT_SECRET. DO NOT SHARE THEM.
If you want to use the Topic Collector API which proxies the old Tag Collector
API, copy your userkey.pem and usercert.pem to $WORKDIR/cmsgit-data.
cp ~/.globus/usercert.pem <USERKEY_PATH>
cp ~/.globus/userkey.pem <USERKEY_PATH>
Finally you need to get an "Personal API Access Token" from github for the user which you consider administrator of your repository (cmsbuild, in the case of the official instance). This can be done by going to https://github.com/settings/applications and clicking on "Create new token".
Create a secrets.py file with the following contents:
cat << EOF > $CMSGIT_ROOT/secrets.py
github_secrets={"production":
{
"client_id": "CLIENT_ID",
"client_secret": "CLIENT_SECRETS",
"admin_auth_token": "ADMIN_TOKEN",
}
}
cern_secrets = {
"usercert": "USERPEM_PATH",
"userkey": "USERKEY_PATH",
"tokens_path": "TOKENS_PATH"
"cache": "CACHE_PATH",
}
EOF
In case of the Official Instance, you can simply copy your secrets from cmsbuild private afs area:
cp ~/private/secrets.py $CMSGIT_ROOT/secrets.py
mkdir -p $WORKDIR/cmsgit-data/secrets
cp ~/.globus/usercert.pem $WORKDIR/cmsgit-data/secrets/usercert.pem
cp ~/.globus/userkey.pem $WORKDIR/cmsgit-data/secrets/userkey.pem
Make sure you cannot access the file from the web and make sure they cannot be read by anyone but your user and the webserver.
Finally we need to create a couple of web writeable directories which will be used to hold cached results (FIXME: use MEMCACHE / real DB).
mkdir -p $WORKDIR/cmsgit-data/tokens
mkdir -p $WORKDIR/cmsgit-data/cache
fs sa -dir $WORKDIR/cmsgit-data/tokens -acl webserver:afs rlwikd
fs sa -dir $WORKDIR/cmsgit-data/tokens -acl system:anyuser none
fs sa -dir $WORKDIR/cmsgit-data/cache -acl webserver:afs rlwikd
fs sa -dir $WORKDIR/cmsgit-data/cache -acl system:anyuser none
All API access is over HTTPS, and accessed from the eulisse.web.cern.ch/eulisse/cgi-bin/git-collector domain. All data is sent and received as JSON. You'll have to be authenticated via CERN SSO to get the data.
Many API methods take optional parameters. For GET requests, any parameters not specified as a segment in the path can be passed as an HTTP query string parameter.
For POST requests, parameters not included in the URL should be encoded as JSON with a Content-Type of ‘application/x-www-form-urlencoded’:
GET /buildrequests
- state: a comma separated list of states the request can be in in order to be listed.
- architecture_match: a regular expression which the architecture of the request needs to match in order to be included in the results.
- release_match: a regular expression which the release name of the request needs to match in order to be included in the results.
Status: 200 OK
[
{
"status": "",
"author": "",
"buildMachine": "",
"pid": "",
"payload": "test",
"lastSeen": "",
"id": "1"
}
]
Gets the list of buildrequests.
GET /buildrequests/:id
Response:
{
"id": "1",
"payload": ""
}
POST /buildrequests
architecture: The architecture of the build request.release: The release name to which the build belongs.repository: The name of the repository to use as a source.PKGTOOLS: PKGTOOLS tag or branch. Optional, will default toIB/<release series>/<architecture>.CMSDIST: CMSDIST tag or branch. Optional, will default toIB/<release series>/<architecture>.ignoreErrors: Whether or not ignore errors. Optional, will default to false.package: The package to build. Optional, defaults to cmssw-tool-conf.continuations: What to build next, in case of success. It takes a comman separated list of : pairs. Optional, defaults to nothing.syncBack: Wether to sync back to the parent repository. Optional, defaults to false.debug: Debug mode. Optional will default to false.
POST /buildrequests
{
"CMSDIST": "IB/CMSSW_6_2_X/stable",
"PKGTOOLS": "V00-21-XX",
"architecture": "slc5_amd64_gcc472",
"continuations": "",
"debug": false,
"hostnameFilter": ".*",
"ignoreErrors": true,
"package": "uuid",
"release": "CMSSW_6_2_X_2013-07-01-1400",
"release_queue": "CMSSW_6_2_X",
"repository": "cms.week0",
"syncBack": true,
"tmpRepository": "eulisse"
}
Status: 200
{
"author": "eulisse",
"url": "",
"lastModified": "2013-07-01T16:54:56",
"pid": "",
"id": "5",
"machine": "",
"state": "Pending",
"architecture": "slc5_amd64_gcc472",
"hostnameFilter": ".*",
"release": "CMSSW_6_2_X_2013-07-01-1400",
"payload": {
"repository": "cms.week0",
"package": "uuid",
"PKGTOOLS": "V00-21-XX",
"ignoreErrors": true,
"CMSDIST": "IB/CMSSW_6_2_X/stable",
"continuations": "",
"syncBack": true,
"architecture": "slc5_amd64_gcc472",
"debug": false,
"release": "CMSSW_6_2_X_2013-07-01-1400",
"tmpRepository": "eulisse"}
}
In case a malformed request is done, it returns:
Status: 400 Bad Request
PATCH /buildrequests/<id>
PATCH /buildrequest/0
{
"state": "Pending",
"url": "http://build-logs.com/0",
"pid": 100
}
GET /cvs-queue
Response:
Status: 200 OK
[
{
"name": "CMSSW_6_2_X"
}
]
GET /externals
where:
Example:
GET /externals
Response:
Status: 200 OK
[
{"ref": "CMSSW_5_3_X"},
{"ref": "CMSSW_6_1_X"},
{"ref": "CMSSW_6_2_X"}
]
GET /externals/<release-queue>
where:
<release-queue>is the release queue for which you want to get all the externals.
Example:
GET /externals/CMSSW_6_2_X
Response:
Status: 200 OK
[{
"CMSDIST_TAG": "IB/CMSSW_6_2_X/stable",
"PKGTOOLS_TAG": "V00-21-XX",
"SCRAM_ARCH": "slc5_amd64_gcc472"
}]
GET /cvs-queue/<release-queue>
Example:
GET /cvs-queue/CMSSW_6_2_X
Response:
Status: 200 OK
{
"131858": [["RecoMET/METFilters", "V00-00-15", "V00-00-11-03"]]
}
GET /cvs-queue/<release-queue>/history
Example:
GET /cvs-queue/<release-queue>/history
Response:
Status: 200 OK
[
{
"id": "133069",
"publisher": ["xxxx"],
"tagset": [["HLTrigger/Configuration", "V14-10-22"]],
"releases": ["CMSSW_6_2_X_2013-06-10-1400"],
"description": ["Comments, as found in TC"]
},
{
"id": "133070",
"publisher": ["xxxx"],
"tagset": [["HLTrigger/Configuration", "V14-10-23"]],
"releases": ["CMSSW_6_2_X_2013-06-11-1400", "CMSSW_6_2_X_2013-06-11-0200"],
"description": ["Comments, as found in TC"]
},
]