API and relationship with rust-codex32

[apoelstra]

I've been thinking about this for a little while, but #95 made me think of it again.

In Bitcoin, we have several checksums that are all degree-2 BCH codes over the the same alphabet:

• The OG bech32 BIP 173
• bech32m BIP 350
• The descriptor checksum BIP 380
• codex32 BIP 93

There are also "blech32" and "blech32m" which are part of Elements/Liquid, and probably others, and probably new ones will pop up as the years ga by.

In all cases, we have the same character set, the same "hrpstring" encoding, basically the same algorithm (although BIP 93 is a bit annoying since, as a 65-bit checksum, we cannot pack intermediate values into u64s) for checksum generation and verification, and basically the same algorithm for error locating/correcting.

Currently this crate provides support for bech32 and bech32m, and deliberately does not expose any internals, which means that we wind up duplicating some code.

I think we should expose some sort of internals module, and some generic code, to make this easier.

Thoughts?

cc @clarkmoody @Kixunil

[clarkmoody]

I think that's a good idea, though maybe behind an expose_internals or advanced feature flag? We can move this functionality into an internals module, with API exposure via flag.

Given that we will probably see more encoding schemes in the future, how does such a feature impact whether this library can stabilize at 1.x?

[clarkmoody]

The sort of flagging I was thinking about would look like this:

#[cfg(feature = "advanced")]
pub use internals;

// Private module without flag
mod internals { .. }

So other encoding protocol developers opt-in to the goodies, but normal consumers like wallet devs don't see the internals as part of a normal dependency.

[apoelstra]

I think this is reasonable, except I think rust-miniscript and rust-codex32 are going to turn this on, to implement their own checksums ... so even "normal dependencies" will wind up having the feature turned on, if they also have a miniscript or codex32 dep.

I don't feel strongly either way, but I think the feature will have limited value because of this.

[clarkmoody]

Sticking this inside a dedicated internals crate is probably enough signal to users that you shouldn't use what's inside unless you really know what you're doing.

Either way, it would be good to go see how downstream dependents are consuming this thing in a few years and let that inform further development.

[Kixunil]

I understood it, my argument still applies: we will document with --all-features for docs.rs so it won't really hide it (no, I don't want to exclude the feature, that'd be awful). There's no point of pushing people away that much. Instead just use this documentation:

Module containing low-level internal types

You shouldn't need this in the vast majority of cases, so don't worry about it.
This module is provided for projects that want to implement their own encoding/checksumming schemes using same primitives bech32 uses.

I doubt anyone will accidentally type bech32::internals::Foo and have it compile.

That being said, I wouldn't mind having a separate crate so that other crates don't need to import whole bech32 impl with it. Although arguably the amount of code will be low.

[apoelstra]

Let's do it all in one crate for now -- I think we've agreed on a module separation, that, if possible, will be really easy to separate out. I suspect that the amount of non-internals code will be so small that this'll be a non-issue.

[Kixunil]

I was thinking it could make sense to make each of these a separate crate:

• base32 - similar API as base64
• bch - the error correction/detection code; name sucks so much it hurts (bitcoin-bch sucks more), I'm up for calling it bose-chaudhuri-hocquenghem
• hrpstring (can we come up with a better name?)

And if not a separate crate then at least try to make the API as if it was a separate crate. I don't have much time to dig into this more deeply.

[apoelstra]

I think it's reasonable to split up the modules "as though" these were separate crates. But I think it'd be too confusing for us to try to actually split them up, especially when the names are so confusing. base32 usually refers to a different base-32 encoding (I'm not sure if there's a "standard" one but e.g. I have a base32 tool on my system that does something that looks kinda like base64; Nix has its own base32 encoding which has existing Rust crates for). Our encoding we refer to as bech32 ... which is unfortunate because that's also the name of the error correcting code.

Then bch sounds too much like bech32, plus it's misleading because we don't intend to support all BCH codes ... just degree-two ones over GF32.

And hrpstring is an obscure term for a very narrow part of what this crate does. I don't think that it's really separate from bech32.

[apoelstra]

So, here's my thought for an API exposure:

• internals
  • fe32 (GF32 elements, i.e. "bech32 characters")
  • fe1024 (GF1024 elements, used in error correction/location, and also used first-class by SLIP39, though they have their own encoding)
  • checksum (generic degree-2 BCH code checksum generation and locating, including traits for people to define their own)
  • correction (error location and correction)
  • hrpstring (checksummed strings of GF32 elements, optionally with a HRP and a 1; this also contains functionality for going from u5 to u8 and back)

Then the root of the crate defines generic functions for "end-to-end" operations like taking data and producing a checksummed hrpstring from it, or an iterator which does the same, etc) and the specific checksums Bech32 and Bech32m.

[apoelstra]

My plan is to just bring in the codex32 fe3::Fe type and expand the ops traits to cover all the combinations of references.

I don't see why we would need traits, other than Into<char> and TryFrom<char>, to express this encoding.

[tcharding]

Ok, I'll keep playing with it. Thanks

[Kixunil]

Yeah, don't overdo it with traits. Do we have a better name than internals since that's a crate bech32 might start depending on at some point? Maybe primitives?

[apoelstra]

primitives sounds good to me. Though I think the internals collision would be okay in practice.

[tcharding]

For what its worth, I like primitives.