Add Submission Process Rules

[fsschneider]

This PR adds a submission process rules document and adapts the rules accordingly.

Before we merge it, this PR should be checked (at least) by @znado, @georgedahl, and @TheKanter to ensure that it satisfies all requirements. That's why I marked this PR as WiP.

---

I generally tried to split content between the scientific part (in the rules document) and the more logistical stuff of the competition process (in the submission process document). Basically, if someone wants to run the benchmark outside of the competition schedule, they would only need the rules doc, but not the submission process rules doc.

Questions:

• @znado, do you have anything to add or modify in the description of the code freeze? In a previous WG meeting, you mentioned a "soft freeze", should we describe this? Should we perhaps add a "we reserve the right to modify the code even after the code freeze deadline if we encounter breaking bugs"?
• Is it a good idea to allow anyone to challenge a submission? We could restrict it to submitters, but perhaps members of our working group want to challenge submissions but not submit themselves.
• I envision that we have a (private) Google sheet with all registered submissions. It contains columns for submission ID, authors, URL to the GitHub repo, checkboxes for the licensing documents, etc. With an expected number of submissions of O(10), this should be doable.
• Do we need a deadline for the submission specifying the batch sizes on the held-out workloads?
• Should we also "freeze the rules" at the same time we freeze the codebase?
• Should the registration of a submission be sent to just the chairs or to the entire working group?
• How exactly does the verification process for results work? Is there a penalty if we cannot verify their score (what's the threshold)?
• Should we reintroduce the statement "only verified results are eligible to win the benchmark and be awarded prize money"? Does this include the "jury award"? How can we formalize that we will ensure that we verify all scores that have a chance at winning prize money?
• I suggested that the results logs should be provided in a separate /results directory within the /submission folder, do you agree?
• Do we want to/need to formalize more how we select promising submissions from the qualification set? At some point, we had a first draft of rules for acquiring compute support.
• Can we specify who the "trusted third party" is that provides the random seed for sampling held-out workloads and hyperparameters?
• I stipulated that the spirit jury will decide by vote, deciding with a simple majority. Is that alright with everyone? Should we formalize this more?
• Do you all agree with the process of how the awards committee decides on the awards?
• @TheKanter, does the section about prize money require further legal details? E.g. how exactly the prize money is paid out, etc.
• @TheKanter, do we need to precisely define what constitutes an institution for the purpose of "ineligibility and conflict of interest"?

Follow-Up ToDos:

• Perhaps add more "sample submissions"?
• Specify who the "trusted third party" is that provides the random seed for sampling held-out workloads and hyperparameters.
• Implement (and link in the text) the function that samples the held-out workloads and hyperparameters.
• Who serves in the spirit jury?
  • It requires expertise in the rules, i.e. a member of the WG.
  • Ideally includes people that are ineligible for prize money.
• Who serves on the awards committee?

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[fsschneider]

@TheKanter this is ready to review for you. Specifically, the last two sections "Awards and prize money" and "Ineligibility and conflict of interest" need legal double-checking.

[fsschneider]

@znado @priyakasimbeg @georgedahl

I believe we can merge this and do any other modifications in a future PR.