automark

Automatic code marking

These scripts will work through a folder of folders containing zipped Java projects and test them using various metrics. The current checks are the following:

1. Code compilation.
2. Code execution.
3. Data output validation (based on random inputs).
4. Indentation correctness.
5. Variable name quality.
6. Code comment quality.

Usage

The minimum requirements to run the batchmark.py script are a task number a path to the folder containing the folders of students' work. The script will then assume default values for everything else.

python batchmark.py <task number> <path to students' work>

For example:

python batchmark.py 1 ./DLJ

There are other parameters for controlling where to find inputs and store outputs. Full usage details can be obtained using python batchmark.py -h at the command line to get the following.

usage: batchmark.py [-h] [-i INITIALS] [-b BUILD] [-t TEMPLATE] [-d DETAILS]
                    [-s SUMMARY]
                    TASK WORK

Batch marker for 4001COMP Java programming.

positional arguments:
  TASK                  Task number (e.g. 1)
  WORK                  Folder containing students' folders (e.g. ./DLJ)

optional arguments:
  -h, --help            show this help message and exit
  -i INITIALS, --initials INITIALS
                        Marker's initials (default Master)
  -b BUILD, --builddir BUILD
                        Folder to output build files to (default ./build)
  -t TEMPLATE, --template TEMPLATE
                        Word feedback sheeet template (default
                        ./feedback_username.docx)
  -d DETAILS, --details DETAILS
                        Excel student details list (default ./4001COMP Marking
                        2014-15.xlsx)
  -s SUMMARY, --summary SUMMARY
                        Summary of marks as a CSV file (default ./summary.csv)

Dependencies

The automark tool uses python 2.7, along with the following additional requirements:

1. Python package xlrd.
2. Python package python-docx.
3. Python package PLY.
4. JDK (javac compiler and VM).
5. Python package SOAPpy if (optionally) using Sphere Engine

It also uses a modified version of plyj, included in the plyext folder. GNU Octave is needed to execute the analysis scripts

Analysis

Scripts for analysing the data generated by the tool, and optimising the parameters used in the scoring calculations, can be found in th analyse folder. These are for use with GNU Octave. The main process is the optimise.m script. Most of the other files are function definitions used during the processing.

The main optimisation process assumes various identically-sized one-dimensional matrics have been defined:

1. comment_orig - The original human-assigned scores for the comments
2. comment_ave - The calculated average distance between comments
3. comment_sd - The standard deviation of the distance between comments
4. indent_orig - The original human-assigned scores for the indentation
5. indent_error - The number of indentation errors

Each entry represents the result output from checking a single source code submission. The values (apart from the human-assigned scores, obviously) can be extracted from the summary.csv file output by the automark tool.