The wrong value is computed in some Wycheproof test cases

[emillon]

Hi,

While trying to run Wycheproof tests against this library, I noted some failures. In particular, tests 68 & 69 compute a wrong result. For example, for test 68, callipyge returns f6ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7f instead of 0900000000000000000000000000000000000000000000000000000000000000.

Here is a test case:

let hexdump a =
  Array.iter (fun b -> Printf.printf "%02x" b) a ;
  print_newline ()

let from_hex s =
  Hex.to_string (`Hex s)

let () =
  let pub_hex = "89e10d5701b4337d2d032181538b1064bd4084401ceca1fd12663a1959388000" in
  let priv_hex = "a0a4f130b98a5be4b1cedb7cb85584a3520e142d474dc9ccb909a073a976bf63" in
  let public = Callipyge.public_key_of_string @@ from_hex pub_hex in
  let secret = Callipyge.secret_key_of_string @@ from_hex priv_hex in
  let out = Array.make 32 0 in
  Callipyge.ecdh_inplace ~out ~secret ~public ;
  hexdump out

FYI, here is a piece of code that computes the same thing with openssl using pyca/cryptography:

import binascii
from cryptography.hazmat.primitives.asymmetric.x25519 import X25519PrivateKey, X25519PublicKey

if __name__ == '__main__':
    pub_hex = "89e10d5701b4337d2d032181538b1064bd4084401ceca1fd12663a1959388000"
    priv_hex = "a0a4f130b98a5be4b1cedb7cb85584a3520e142d474dc9ccb909a073a976bf63"
    public_key = X25519PublicKey.from_public_bytes(binascii.unhexlify(pub_hex))
    private_key = X25519PrivateKey._from_private_bytes(binascii.unhexlify(priv_hex))
    shared = private_key.exchange(public_key)
    print(binascii.hexlify(shared).decode())

With openssl 1.1.1a, this one computes the correct value.

See openssl/openssl#6687 for a related failure in openssl itself.

Thanks!