Assurance Subject Matter Dimensions

[chrisdburr]

In the TEA methodology, goals may refer to normative principles such as fairness or interoperability. The process of developing an argument and grounding it in evidence helps to operationalise the principle and add further specificity. This is important in cases where the goal can be highly ambiguous.

The CDEI proposes the following dimensions (focused on what they call ‘Assurance Subject Matter’):

The chapter explaining the diagram in more detail is here: https://cdeiuk.github.io/ai-assurance-guide/subject-matter

In short, the dimensions are as follows:

1. Unobservable vs Observable: This dimension refers to whether the subject matter can be directly seen or measured (observable) or if it must be inferred through proxies (unobservable). For instance, the accuracy of an AI system can be directly tested and observed, while potential societal impacts are unobservable and need to be inferred.
2. Subjective vs Objective: This indicates whether the assessment or measurement of the subject matter is reliant on personal interpretations and judgement (subjective), or if it’s based on clearly defined and quantifiable standards (objective). An example could be choosing a definition of fairness for an AI system, which is subjective, versus measuring the system’s false positive rate, which is objective.
3. Ambiguous vs Explicit: This relates to the clarity of the subject matter, with explicit subject matters having a clear consensus on assessment criteria, while ambiguous ones require more interpretation and expert judgement. Determining potential societal impacts of AI use might be ambiguous and require judgement, while measuring its accuracy is explicit and can be standardized.
4. Uncertain vs Certain: This refers to the level of certainty with which the outcomes or risks of a subject matter can be known, from being definite and predictable (certain) to being unknown or having a significant degree of unpredictability (uncertain). The potential risk of an AI system causing harm in the future is uncertain, while the system’s performance on a specific task can be known with certainty.

Are these dimensions useful for this methodology and platform? Should we incorporate them into our guidance?