OCAv1 design choices

[blelump]

The recent ORFG meetings have shown uncertainties and ambiguities in understanding the rationale behind particular design choices made in OCA v1.

1. OCA Canonical form. We defined (currently missing in the spec!) the OCA canonical form as a synthesis of human readability and machine actionability. OCA Bundles tend to be read from time to time for various reasons. We, therefore, introduced custom ordering into the Bundle, purposely ignoring existing standards like https://datatracker.ietf.org/doc/rfc8785/ . We furthermore found Sam Smith's concept (introduced in KERI events) to start with the version attribute (usually serialized to v) incredibly useful in OCA-related concepts like the OCA Bundle. Specifically, starting with v enables the pre-validation of the Bundle, even without deserializing the JSON. It is possible because v is a fixed length attribute that contains not only a version but also the length of the content. We, therefore, always know where to look for the version upon which we can employ proper parsing mechanics (because v always starts the JSON and is fixed length). Next to v is the digest (d) of the Bundle. On the one hand, having it at the beginning of the JSON is convenient because it's the most "misleading" attribute, precisely its value (just looking at it can tell us whether it's the same Bundle or something different). On the other hand, having it exactly right after v enables us to run optimizations when calculating the SAID. It is doable because v is a fixed length, so d is also a fixed length. When serializing to JSON, we can replace certain characters in plain old string by replacing specific bytes. Such an approach is crucial for improving serialization speed. Except for the v and d attributes, OCA serialization is compatible with the aforementioned JSON Canonicalization Scheme.

2. SAID, SAD, and CESR. SAD is the document, i.e., OCA Bundle. SAID is the self-addressing identifier of a SAD. SAID conceptually is part of CESR (i.e. code tables) that works with SADs (therefore, distinct spec). The SAID/SAD duo gives the unambiguous recipe for calculating the document identifier, that is, its SAID. To quote the spec: To clarify, a SAID is different from a standard content address or content-addressable identifier in that a standard content-addressable identifier may not be included inside the contents it addresses. This is the SAID novelty that no other concept addresses. The other reason OCA uses SAIDs is the compatibility of the HCF ecosystem. Since 2020, we have been developing https://dkms.colossi.network/, which is the KERI protocol under the hood. KERI is Sam Smith's invention, and understandably, it also consumes his other concepts, such as CESR and SAID.

   SAID, furthermore, is forward-compatible with hashing algorithms that have yet to be established or invented via its code tables. OCA, in this regard, must also be forward-compatible. We are specifically interested in the first and second pre-image resistance for any hashing algorithm in OCA, and historically, MD5 and SHA1 were found to lack these properties.

   OCA doesn't specifically require full-fledged CESR support. It implicitly uses a subset of it by employing SAID. SAID requires a code table from CESR and encoding mechanics. The benefits of using full-fledged CESR (and what CESR is) are out of the scope of this discussion, but they are definitely use-case-specific. For an idea, see Extracting s from ACDC into CESR-based protocol WebOfTrust/ietf-cesr#45 .

As of 2024, most concepts from Sam Smith's tech stack have been fully described in specifications. As stated above, OCA has greatly benefited from these concepts and has applied them to solve various problems. We also purposely don't describe some of the mechanics in OCA spec because they're already documented in other specs, especially those related to SAID. Finally, we're always open to considering new concepts for the next major release. If there's a concept that outperforms Sam's invention in any way, we'd love to hear about it.

Edit: Elaborate more on CESR.

[carlyh-micb]

"CESR looks complex because it's a clever protocol. SAID isn't technically CESR but uses CESR concepts, i.e., the code table and the encoding process described in the #58 (comment) above."

I'm a big fan of clever, but a simple to implement without additional complexity is going to win. Ease of adoption is a winner over clever.

There are two approaches to make clever more easy to adopt. Write it out better making the line of uptake clear and easy without having to sift through tons of documentation or pick something already more accessible which meets the main requirements. Both solutions have benefits and trade-offs.

I would love to have it explained why CESR is so clever for schema digests. People who aren't coders are going to be making business decisions about this and there should be something compelling for the complexity. What is so compelling? I believe there is something here, but I would like to hear it articulated.

[blelump]

CESR isn't required for OCA; SAID is. CESR is clever because it addresses issues in the so-called SSI space precisely:

• how easy it is to attach the digital signature to the content it signs
• seamless switching between text and binary
• framing for stream processing

This is the full-fledged CESR, but OCA neither uses it nor requires it. All it needs is SAID. See WebOfTrust/ietf-cesr#45, where OCA could use the actual full-fledged CESR, but it is use case-specific. The whole authenticity layer, so when https://dkms.colossi.network/ comes into play, uses CESR for streaming by default. However, for the end user, it's an implementation detail that she cannot even notice.

Business decisions shall not be based on what is CESR or what's inside it but on how the overall technology solves a given problem. OCA takes SAID from CESR, and SAID has specific characteristics (discussed above) from which the business implicitly benefits. Where the business really benefits is from the so-called OCA Ecosystem, which includes the tooling and products around OCA.

For the record, CESR spec

[swcurran]

We don’t need to understand the design decisions. I stated I would use different ones, but so be it and I’m fine if you don’t like those decisions.

What is needed is a clear definition in the spec. of how to generate a digest, and how to verify it. The current definition saying “use a SAID” and pointing to the expired SAID spec is not working.

[blelump]

We don’t need to understand the design decisions. I stated I would use different ones, but so be it and I’m fine if you don’t like those decisions.

The spec authors have the right to explain the current version's design choices and defend them in public. As stated in the last paragraph, we can constructively discuss alternatives / other options based on the assumptions defined above or redefine them if they are wrong. OCA emerged to address the biggest topic in the digital space; therefore, there must be trade-offs to find the common denominator.

What is needed is a clear definition in the spec. of how to generate a digest, and how to verify it. The current definition saying “use a SAID” and pointing to the expired SAID spec is not working.

Fair argument. This is the superior spec.

[swcurran]

Sorry — you are right, you are welcome to defend the choices and it can be helpful. I like that you have done it in an issue, and not put it in the spec.

Here is my try at explaining the process — based on that section of the CESR spec, pulling together the different parts:

From the JSON example, and then jumping back to the Python example, reading the order preserving section, and using the master code table.

• Start with the JSON containing a SAID field.
  • The producer MUST NOT change the JSON item ordering between calculating the SAID and placing it into an overlay.
  • The verifier MUST preserve the JSON item ordering when receiving the overlay being verified.
• Find the sting value of the SAID item — base_capture in the Base Capture overlay, digest in all other overlays.
• Define (producer) or extract (verifier) the first character of the SAID string.
  • Question: Is there more than the first character involved in the prefix or are there relevant parts of this to be understood?
• Use this table to define/determine the hash algorithm being used and accordingly, the length of the SAID string.
  • The verifier MAY use the length of the value vs. looking it up in the table.
  • Question: Is it safe to just use the hash algorithms subset of the table? Are other parts used?
  • Question: Does/will the OCA spec limit the permitted hash algorithms, or must all be supported by all implementations?
• Set the string value of the SAID item in the JSON to length of hash “#” characters.
• Remove all external whitespace from the JSON.
• Calculate the hash of the JSON using the defined/calculated hash algorithm
  • Question: Is there any encoding of the hash into characters or is that implied in the algorithms?
  • Question: Is there padding involved in calculating the hash string?
• Prefix the hash with the defined/extracted first character of the SAID item value
• The producer of the SAID sets the value of the SAID item to the resulting string.
• The verifier of the SAID checks that the value of the SAID item from the JSON matches the calculated string.

Is there anything missing?

A critique of the CESR spec, that perhaps could be addressed in the OCA spec, is the process for going from the raw hash output to the prefixed string.

[carlyh-micb]

I think there is something missing here Stephen, it is not enough to calculate the hash, it also has extra post-processing steps. Those are part of the KERI SAID specification. I am working with Kent Bull of KERI to better document this.

[carlyh-micb]

If I need to defend the choice of spec then I need to be able to defend, or at least understand design decisions. Anyone picking this, and then having to explain why it takes much more developer time to develop because of design choices will need to defend this choice.

Sam Smith's specification is not finalized but a work in progress undergoing active development, is it even versioned at this point? Why not put it in OCAv3? Then OCAv2 would be using standards widely adopted (canonicalize using JSON canonicalization, hash using multihash). Version 3 can come out virtually the same but with the fancy new CESR support (and even come out quickly), version 2 can be used immediately with documented tools.

[carlyh-micb]

Checking in on canonicalization -

OCA schema orders everything according to JSON canonicalization method, except it pulls out two special terms, v and d.

The v is the version, which isn't present in the .zip schema bundle, and the d is the digest (also represented differently in the .zip schema bundle because it isn't called d there).

Does anyone know how JSON canonicalization handles words like: ᐃᓄᒃᑎᑐᑦ

Can anyone post an example JSON only, including the v and d values, OCA schema?

As a reminder, here is the JSON code of a Capture Base from the .zip schema bundle (no d or v)

Is type also privileged like v and d?

{
  "type": "spec/capture_base/1.0",
  "digest": "EALZtciYP_Tn97BFilKL3C8ofMxF-xrswDmU9NC4CP9M",
  "classification": "RDF401",
  "attributes": {
    "collect_date": "DateTime",
    "collect_time": "DateTime",
    "date_in": "DateTime",
    "date_out": "DateTime",
    "duration": "DateTime",
    "farm": "Text",
    "final_mass": "Numeric",
    "initial_mass": "Numeric",
    "latitude": "Text",
    "longitude": "Text",
    "moisture": "Numeric",
    "paddock_id": "Numeric",
    "sample_id": "Text",
    "sample_type": "Text",
    "samplers": "Array[Text]",
    "temperature": "Numeric",
    "time_in": "DateTime",
    "time_out": "DateTime"
  },
  "flagged_attributes": [
    "farm",
    "latitude",
    "longitude",
    "samplers"
  ]
}

[carlyh-micb]

ChatGPT says

The word "ᐃᓄᒃᑎᑐᑦ" will be sorted according to the Unicode values of its characters, and its position in a list would be determined based on those code points. The sorting is lexicographical based on the Unicode values, starting with the first character and proceeding through the string.

From a list of 1, 003, ball, ᐃᓄᒃᑎᑐᑦ this is how it would be sorted:

"003" (smallest code points starting with U+0030)
"1" (next, U+0031)
"ball" (next, starting with U+0062)
"ᐃᓄᒃᑎᑐᑦ" (largest, starting with U+1403)

[blelump]

Is type also privileged like v and d?

yes it is, for the same reason.

Does anyone know how JSON canonicalization handles words like: ᐃᓄᒃᑎᑐᑦ

The same as any other UTF-8 encoded byte vector. UTF-8 is a table of characters where each gets a unique byte[] value. If the ordering is given in ascending order, the words will be sorted according to their byte[] value. Smaller values go first, and larger last.

[carlyh-micb]

And just for completeness, I asked ChatGPT about ordering which might be obvious for coders but not for me.

Objects sorted by UTF-8 byte values will be the same as if sorted by Unicode code points.
Explanation:

UTF-8 is designed to ensure that the byte order of the encoded characters corresponds to the order of their Unicode code points. This means that when you sort strings by their UTF-8 byte sequences, you are effectively sorting them by their Unicode code points as well.
How It Works:

Single-byte Characters (e.g., ASCII):
    ASCII characters (Unicode code points U+0000 to U+007F) are encoded as single-byte values in UTF-8. Their byte values directly correspond to their code points.
    Example: "A" (U+0041) is 0x41 in both Unicode code point and UTF-8.

Multi-byte Characters:
    For characters with higher code points, UTF-8 uses multi-byte sequences, but the bytes are ordered in such a way that the UTF-8 byte sequences maintain the same order as their Unicode code points.
    Example:
        "A" (U+0041) → 0x41 (1 byte)
        "©" (U+00A9) → 0xC2 0xA9 (2 bytes)
        "漢" (U+6F22) → 0xE6 0xBC 0xA2 (3 bytes)

Key Point:

Lexicographical Order: UTF-8 is designed so that sorting by byte sequences will give the same lexicographical order as sorting by Unicode code points. This property is one of the reasons UTF-8 is so widely used, especially for text processing and storage.

Conclusion:

Sorting objects (like strings) by their UTF-8 byte values will yield the same order as sorting by their Unicode code points. This is a key feature of UTF-8 that ensures consistent and predictable ordering across different systems and platforms.

[blelump]

The actual order is less relevant than the determinism of the process. Therefore, if the ordering produces a consistent order every time, this is good enough.

[carlyh-micb]

Yes. And the order process must be accurately documented so that it can be reproduced.