-
Notifications
You must be signed in to change notification settings - Fork 249
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add serde to Field #506
base: master
Are you sure you want to change the base?
add serde to Field #506
Conversation
This is highly subject to debate. |
BTW how do people deal with this in projects that run in prod? We chose to use serde_with everywhere but I'm thinking of just forking arkworks with this branch and using this atm. |
Thanks for the PR! Couple of points
|
Hm I had something more like this in mind: // in ark-serialize
pub type Compressed<T: CanonicalSerialize + CanonicalDeserialize>(T);
pub type Uncompressed<T: CanonicalSerialize + CanonicalDeserialize>(T);
#[cfg(feature = "serde")]
impl<T: CanonicalSerialize> Serialize for Compressed<T> {
}
#[cfg(feature = "serde")]
impl<T: CanonicalDeserialize> Deserialize for Compressed<T> {
}
#[cfg(feature = "serde")]
impl<T: CanonicalSerialize> Serialize for Uncompressed<T> {
}
#[cfg(feature = "serde")]
impl<T: CanonicalDeserialize> Deserialize for Uncompressed<T> {
} You can use this in your code as follows: #[derive(Serialize, Deserialize)]
pub struct Proof {
pub type: Compressed<Whatever> // or Uncompressed<Whatever>
} Or like this, at serialization/deserialization boundaries: #[derive(CanonicalSerialize, CanonicalDeserialize)]
pub struct Proof { ... }
fn broadcast_to_network(p: Proof, r: &mut Read) {
Compressed(p).serialize_to_json(r); // or whatever is your serde-interfacing code
} If the latter works for your code, I think it's better because it requires minimal code in arkworks libraries. |
Humm, I'm not sure I understand your example. What is Also, if I understand correctly:
As in, if I have the following: pub struct Proof {
pub single: Compressed<Field>,
pub vec: Vec<Compressed<Field>>,
} then I'd have to do things like |
The idea is to define Regarding the two options, you can derive |
hummm I see, but what is the point of Uncompressed? |
Ah We need these additional wrapper types because a blanket |
Do we ever need to send an uncompressed point through the wire? |
People definitely use it when serializing to disk. |
Sorry for the delayed replayed. If I understand correctly this means I will be able to write code like this: #[derive(Serialize, Deserialize)]
#[serde(bound = "T: CanonicalDeserialize + CanonicalSerialize")]
pub struct Proof<T> where T: CanonicalDeserialize + CanonicalSerialize {
pub single: Compressed<T>,
pub vec: Vec<Compressed<T>>,
} It is annoying to have to bind The other problem is that you'll have to dereference with EDIT: I just tried implementing this in our codebase and it's becoming quite a nightmare to track :D Ideally I want to just be able to write #[derive(Serialize, Deserialize)]
pub struct Proof<T> {
pub single: T,
pub vec: Vec<T>,
} and be able to instantiate |
@mimoo The usage I had in mind was more along the following lines: #[derive(CanonicalSerialize, CanonicalDeserialize)]
pub struct Proof {
// Your actual proof contents
}
// elsewhere, in your serialization code
let proof = Pickles::prove(stuff);
let compressed = Compressed(proof);
compressed.serialize_json(); // or whatever else serde method you want to use.
// similarly, while deserializing:
let compressed: Compressed<Proof> = deserialize_json();
let proof = compressed_proof.0;
// rest of your code. I think that should be easier to handle, no? |
oh, but that means we have to implement |
You wouldn't need to impl #[derive(Serialize, Deserialize)]
pub struct Transaction {
state_root: [u8; 32],
signature: [u8; 64],
proof: Compressed<Proof>,
}
#[derive(CanonicalSerialize, CanonicalDeserialize)]
pub struct Proof {
// proof contents
} That is, you can be as granular as you want, and insert the CanonicalSerialize → Serialize transition wherever you want in the stack. |
I see, another problem I can see is that I'm assuming the derivers don't come with the helpers the serde ones come with. For example It doesn't seem like it plays well with unbounded type parameters also: |
We can add a |
As you need wrappers anyways, I'd suggest this become a separate I'd also suggest the verb-ish form |
I'm running into another issue now: I can't nicely JSON serialize some of the types in arkworks. For example, a Radix2EvaluationDomain is just going to look like a bytearray instead of a nice JSON struct. I talk more about this here: o1-labs/proof-systems#1044 I think there's no good solution here besides implementing serde for these types of structs. I still don't understand why we don't do this, and just choose compressed format as the default for elliptic curve points. PS: at this point I'm considering creating a fork called arkworks_with_serde lol |
Why is this a problem? Users cannot read finite field elements anyways.
Why do so many serialization formats exist? Why is serde never going to be part of std? It's total chaos.. Inconsistent, incorrect, or inadequately defined behavior, incorrect canonicalization, format changes, semver fights, poor memory usage on HSMs, etc. Are capnproto arenas using unsafe or C code? Voila, secret leakage risk in serde! It's basically universal that cryptography libraries only ever serialize as bytes, even when they support serde. Afaik even JS crypto libraries and the Web Crypto API obey this convention. Also, |
there are more than just fields in this struct. In our case we'd like to see a human-readable value for
I don't agree, serde is pretty commonly accepted everywhere in the Rust ecosystem, most libraries have a |
Just fyi, I wrote wrapper types like this for arkworks-scale which lives at https://github.com/w3f/ark-scale It's simpler than serde because parity-scale-codec works almost exactly like arkworks' serialization conceptually. It's straightforward to serialize directly from an iterator into the bytes that'll decode as a |
Appears serde_derive is a precompiled binary from someone who works for Palantir. https://twitter.com/eddyb_r/status/1693198713286541429 |
Hello there just to say we are at zksummit and still thinking about this issue. Hope you guys are all well |
Trying to be constructive, here's what it seems to be needed to move forward that issue: Add an optional @mimoo want to take a stab at this with me? |
sorry I have no time on my hands atm :o |
I presume this is no good, but let's start a discussion o.o
It is currently really painful to use arkworks in real-world projects because as soon as you want to serialize to disk or to send something over the wire, you can't use serde. Or at least, you have to resort to very verbose tricks (using serde_with, for example) to be able to serialize/deserialize any struct you have that contains
Field
.