Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add field for data check output to result schema #450

Open
jeanetteclark opened this issue Aug 28, 2024 · 2 comments
Open

add field for data check output to result schema #450

jeanetteclark opened this issue Aug 28, 2024 · 2 comments

Comments

@jeanetteclark
Copy link
Collaborator

jeanetteclark commented Aug 28, 2024

after discussing with @robyngit, we determined that it would really be necessary to filter results from data quality runs by file. This isn't necessarily MVP for the data suite but I think it's close. The easiest way to do this I think would be to add a field to the result schema within run. This would allow us to continue to us the output field to give dataset-level results (eg: 8 files had congruent file types, 1 file was mistyped), in addition to file level results (eg: file.nc appears to be of type application/netcdf but is documented as application/octet-stream).

    <xs:sequence>
      <xs:element name="id" type="xs:string"/>
      <xs:element name="timestamp" type="xs:dateTime"/>
      <xs:element name="objectIdentifier" type="xs:string" minOccurs="0"/>
      <xs:element name="suiteId" type="xs:string" minOccurs="0"/>
      <xs:element name="result" type="tns:result" minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
  </xs:complexType>

proposed new schema for result could look something like this (though I'm very much not sold on "dataOutput" as the element name, it's a placeholder.

  <xs:complexType name="result">
    <xs:sequence>
      <xs:element name="check" type="tns:check" minOccurs="0"/>
      <xs:element name="timestamp" type="xs:dateTime" minOccurs="0"/>
      <xs:element name="output" type="tns:output" nillable="true" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element name="status" type="tns:status" minOccurs="0"/>
      <xs:element name="dataOutput" type="tns:dataOutput" nillable="true" minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="dataOutput">
    <xs:simpleContent>
      <xs:extension base="xs:string">
        <xs:attribute name="id" type="xs:string" use="optional"/>
      </xs:extension>
    </xs:simpleContent>
  </xs:complexType>

thoughts welcome @mbjones when you return

@jeanetteclark
Copy link
Collaborator Author

jeanetteclark commented Sep 4, 2024

Since the output field is repeatable, so we just need to add id as an optional attribute.

@jeanetteclark
Copy link
Collaborator Author

this is done on the feature-hashstore-support branch, awaiting review

see src/main/resources/schemas/schema1.1.xsd

also see edu.ucsb.nceas.mdqengine.dispatch.Dispatcher line 174 for how an array coming back from python checks is handled (warning: its a little ugly right now)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Review
Development

No branches or pull requests

1 participant