-
Notifications
You must be signed in to change notification settings - Fork 34
Manage Tester
A tester system is one where the assignment testing occurs, most often by running a virtual machine and testing the assignment inside that. Assignments are sent from a storer system, they are placed in a queue for the tester and then picked up from the queue and tested/checked (inside an environment such as a virtual machine). Results from the testing/checking phase are sent out back to the storer.
A tester system for vmchecker consists of two sets of components: executors and queue managers.
An executor is a process that starts up the testing environment (most often a virtual machine), grabs an assignment and a test archive, kick off the checking phase and grabs back the results. An executor manages a given kind of environment (for example VMware, VirtualBox, KVM and so on). There is usually only a single executor type on a given tester system, for a particular type of environment (i.e. virtual machine implementation/hypervisor).
There is a queue manager for each class. Each class uses a Unix account and stores information about the executor and queue manager. A queue manager would handle check requests for a given class and kick of the executor for the class virtual machine.
Each class requires a Unix account. Inside the Unix account one stores the folder structure for queue managers and virtual machines for the class. Typically you would store the queue manager configuration in a tester/ subfolder and the virtual machines in a vm/ subfolder. A sample folder listing (for the asm account on the checker.cs.pub.ro tester system) is shown below:
asm@checker:~$ tree --charset ASCII -L 3 -F
.
|-- tester/
| |-- callback_sftp.log
| |-- config
| |-- queue/
| |-- queue-manager.log
| `-- tmpunzip/
`-- vm/
`-- 2015/
|-- iocla-linux/
`-- iocla.zip
In the folder listing, the vm/ subfolder stores the iocla-linux/ virtual machine and the tester/ subfolder stores the queue manager configuration files:
-
configis the actual configuration file for the queue manager - the
queue/subfolder is the assignment queue; assignments sent out from the storer are placed in this folder - the
queue-manager.logfile stores logging messages output by the queue manager - the
tmpunzip/subfolder is used for unpacking the current assignment - the
callback_sftp.logfile stores logging information from when results from an assignment are sent back to the storer system
We assume you've already setup an Unix account for the storer system and for the tester system and you've made SSH public key-based authentication available, by following the instructions here.
On the tester Unix account, create a folder for the virtual machines (we recommend naming it vm/) and copy and unpack the virtual machine(s) in that folder. Important: If using VMware these virtual machines need to have VMware Tools installed. You will then fire up vmware or ask the administrator for that to add a snapshot for the virtual machine. The path to the virtual machine is to be used in the virtual machine configuration section on the storer.
In order to create the folder for the queue manager, you would use the vmchecker-init-course command as shown below.
user@checker:~$ mkdir tester
user@checker:~$ cd tester/
user@checker:~/tester$ vmchecker-init-course tester
A new default vmchecker configuration file was written to /home/asm/tester/config. Update it before running any vmchecker code.
INFO:vmchecker.initialise_course: -- tester init done setting up paths and db file.
user@checker:~/tester$ ls -l
total 16
-rw-r--r-- 1 user user 7520 Dec 28 19:21 config
drwxr-xr-x 2 user user 4096 Dec 28 19:21 queue
drwxr-xr-x 2 user user 4096 Dec 28 19:21 tmpunzip
The tester/ subfolder now stores the required files for running the queue manager.
Before running the queue manager, we need to configure it by updating the tester/config file. In that file you need only update the first attributes; an example of configuring them is shown below:
asm@checker:~/tester$ head config
[vmchecker]
root = /home/asm/tester
repository = /home/asm/vmchecker-storer/repo
CourseName = Introducere in organizarea calculatoarelor si limbaje de asamblare
[...]
In the configuration above:
-
rootis the path to the queue manager folder on the tester -
repositoryis the path to the class folder on the storer -
CourseNameis the name of the class
The following actions need to be handled by the privileged user (i.e. the user that configure vmchecker support on the tester system).
The first thing is to mention support for the new class in vmchecker. You do that by adding a new entry in the /etc/vmchecker/config.list file. The file is simple: each line consists of the class identifier followed by a colon (:) and by the path to the queue manager configuration file. Below is a snippet of the file:
root@checker:~# cat /etc/vmchecker/config.list
SO:/home/so/tester/config
APD:/home/apd/tester/config
AC:/home/ac/tester/config
[...]
The you need to fire up the queue manager. We recommend you use supervisord for that. You install supervisord using:
# apt-get install supervisor
and you configure vmchecker support for queue managers in /etc/supervisor/vmchecker.conf. In the vmchecker.conf file you add entries for each queue manager (i.e. for each class) similar to the sample below:
root@checker:~# cat /etc/supervisor/conf.d/vmchecker.conf
[program:vmchecker-iocla]
command=/usr/local/bin/vmchecker-queue-manager --course_id=IOCLA --stdin=/dev/null --stdout=/home/asm/tester/queue-manager.log --stderr=/home/asm/tester/queue-manager.log
directory=/srv
autostart=true
autorestart=true
startretries=3
stderr_logfile=/dev/null
stdout_logfile=/dev/null
user=asm
[program:vmchecker-so]
command=/usr/local/bin/vmchecker-queue-manager --course_id=SO --stdin=/dev/null --stdout=/home/so/tester/queue-manager.log --stderr=/home/so/tester/queue-manager.log
directory=/srv
autostart=true
autorestart=true
startretries=3
stderr_logfile=/dev/null
stdout_logfile=/dev/null
user=so
[...]
This is how you configure the running of the queue manager and the redirection of standard output and standard error to the queue-manager.log file for each class account.
After adding an entry you run the commands below to start and check the queue manager:
root@checker:~# supervisorctl reread
vmchecker-uso: available
root@checker:~# supervisorctl update
vmchecker-uso: added process group
root@checker:~# supervisorctl status
vmchecker-iocla RUNNING pid 21350, uptime 0:17:19
vmchecker-so RUNNING pid 21752, uptime 0:08:22
vmchecker-so2 RUNNING pid 22035, uptime 0:02:26
vmchecker-uso RUNNING pid 22095, uptime 0:00:05
The output of the commands above is just sample output.
Now you have the queue manager started for the current class. It will grab assignments received from the storer, store them in the queue/ subfolder, fire up the executor and copy the results back to the folder.
In case of any problem when checking assignments, you will want to troubleshoot the queue manager. First thing is to see whether it is started. You do that by either using ps or supervisorctl (as administrator):
root@checker:~# ps -ef | grep IOCLA
asm 21350 21104 0 12:00 ? 00:00:00 /usr/bin/python /usr/local/bin/vmchecker-queue-manager --course_id=IOCLA --stdin=/dev/null --stdout=/home/asm/tester/queue-manager.log --stderr=/home/asm/tester/queue-manager.log
root 26620 26250 0 19:54 pts/1 00:00:00 grep IOCLA
root@checker:~# supervisorctl status vmchecker-iocla
vmchecker-iocla RUNNING pid 21350, uptime 7:53:58
In case the process is not started you need to start/restart it using the supervisorctl command (as administrator):
root@checker:~# supervisorctl start vmchecker-iocla
root@checker:~# supervisorctl restart vmchecker-iocla
In case it started but something else is wrong you should check the logging information in the queue-manager.log file in the tester/ folder:
asm@checker:~$ tail -f tester/queue-manager.log
PUTTING: local:[/home/asm/tester/tmpunzip/vmchecker-he87iC/build-stderr.vmr] remote:[/home/asm/vmchecker-storer/repo/tema2/[...[/results/build-stderr.vmr]
PUTTING: local:[/home/asm/tester/tmpunzip/vmchecker-he87iC/vmchecker-stderr.vmr] remote:[/home/asm/vmchecker-storer/repo/tema2/[...]/results/vmchecker-stderr.vmr]
PUTTING: local:[/home/asm/tester/tmpunzip/vmchecker-he87iC/run-stdout.vmr] remote:[/home/asm/vmchecker-storer/repo/tema2/[...]/results/run-stdout.vmr]
PUTTING: local:[/home/asm/tester/tmpunzip/vmchecker-he87iC/grade.vmr] remote:[/home/asm/vmchecker-storer/repo/tema2/[...]/results/grade.vmr]
STORER CALL program [vmchecker-update-db --course_id=IOCLA --user=theodor.stoican --assignment=tema2]
Cleaning "/home/asm/tester/tmpunzip/vmchecker-he87iC"
Removing job from the queue
Waiting for the next job to arrive
No stale jobs in queue dir "/home/asm/tester/queue"
Waiting for the next job to arrive
The logging message will provide you with insight regarding the inner workings of the queue manager and the problems encountered. Possible issues may be:
- incorrect configuration for public key-based SSH connection between the storer and the tester
- incorrect configuration in the
configfile on the tester system - incorrect configuration in the
configfile on the storer system - incorrect configuration of the virtual machine: incorrect virtual machine path, absence of VMware Tools, no snapshot
- incorrect permissions somewhere in the local file system layout for the tester Unix account; most often some files are owned by
root