Supporting both a public and a private test suite for graded assignments

[m0n0chr0m3]

Context

For one of our courses we're looking for tools to assist us with (graded) assignments. The tool should be able to accomplish these tasks:

1. authenticate users as course members,
2. show the course members their assignment (in the form of marked-up text),
3. provide course members with the means to input a solution (in the form of source code) which gets saved to their user account,
4. automatically run some unit tests on proposed solutions (to see whether the solution's overall form appears to be correct),
5. show course members the results of the unit tests (for their latest submission, but they should be able to inspect the results of their previous submissions as well),
6. once the course member chooses to submit their solution for evaluation: automatically run a second test suite whose results are not shared with the course member, but are anonymized and submitted (together with the course member's submission itself) to the teaching staff for grading.

Evidently, Dodona supports all tasks apart from the last. However, that last task is ostensibly only a slightly tweaked version of the features already offered by Dodona. Furthermore, Dodona already offers tools for grading assignments, though not by making use of a second, private test suite. Finally, we already use Dodona for this course. All in all, it looks like Dodona is very close to being the ideal tool for us.

Request

Are there any plans for Dodona to support such pairs of "public"/"private" (or "visible"/"hidden", or "student-visible"/"teacher-visible") test suites? Or, in case I missed it, does Dodona already offer the features?

[pdawyndt]

Dodona does support “hidden tabs” in its feedback table. Tests in these tabs are only visible to course admins (teachers), not to students. Along the same principle, tests also have “hidden messages” that are only visible to teachers, not to students. It is then up to the judge used for automated assessment to support these hidden tabs/messages, and for authors of programming exercises to specify when they are used in the test suite of the exercise.

As an alternative approach, Dodona also supports re-judging submissions. This gives the option to add test cases after the deadline: re-judging submission will then take into account these new test cases. This might be more confusing for students, as submissions that were evaluated as “correct” at the time a test was taken, could become “wrong” after re-judging (technically the opposite is also possible in case all previous test cases were replaced with new ones, or if submissions/tests might have non-deterministic behavior).

[m0n0chr0m3]

Ah, I see. That resolves my issue, so I'll close the topic.

For future reference for others coming across this topic: as shown in the JSON schema for the output of a judge linked from the judge documentation, tabs have a property permission, which can be set to either "student", "staff", or "zeus". You may need to adapt the judge you use slightly, since the judge needs to known which tabs you want to hide.

Thanks!