React-native bigint64 deserialization problem - refactoring/bigint64le

[Admdebian]

I think react-native can't deserialize prices correctly on all devices.

The price is deserialized to its BigInt32 Little Endian value instead of BigInt64 Little endian value.

I wrote a new code to test this problem.

Example on testnet:

• price public key: 7VJsBtJzgTftYzEeooSDYyjKXvYRWJHdwvbwfBvTg9K
• product public key: GcnU8V2WKq5QXLhJwBQdt2ankU6GRGh6b1bLCxdykWnP
• price buffer: b813fdb904000000341a320c00000000010000000000000056da4b0600000000
• correct value from new code: 20300239800
• incorrect value from current code: 3120370620

Test deserialization (only for unsigned): https://asecuritysite.com/encryption/numbers01

Info from my package.json:

"@babel/core": "^7.12.9"
"@babel/runtime": "^7.12.5"
"@pythnetwork/client": "^2.5.1"
"react": "17.0.2"
"react-native": "0.66.1"
"buffer": "^6.0.3"
"typescript": "^4.3.2"
 [...]
"resolutions": {
  "@types/react": "^17"
}

My node version is: v16.13.0

I have a shim configuration to enable bigint support in react-native environment:

// https://github.com/facebook/react-native/issues/28492#issuecomment-824698934
if (typeof BigInt === 'undefined') global.BigInt = require('big-integer')

[jayantk]

hey @Admdebian thanks for contributing. Definitely would be great to fix this code so it works on react-native. Do you understand why the old code fails on react-native? I'm a bit hesitant to accept a fix without understanding what the problem is.

I actually find the bug quite mysterious, because your implementation of readBigUInt64LE looks like it should do exactly the same multiplications as the existing code.

[jayantk]

I think this code loses precision -- this is a floating point number, which has fewer bits of precision than a 64 bit integer (since some of its 64 bits are allocated to the exponent). I think if you make this a BigInt right off the bat, it should be fine.

[jayantk]

typo in the name. also, i believe this code expects buffer to have 8 bytes storing the number, so you can get rid of the min here and also the Number.MAX_VALUE / maxIndex check below.

[Admdebian]

hey @Admdebian thanks for contributing. Definitely would be great to fix this code so it works on react-native. Do you understand why the old code fails on react-native? I'm a bit hesitant to accept a fix without understanding what the problem is.

I need your help to understand and find the best solution.

I actually find the bug quite mysterious, because your implementation of readBigUInt64LE looks like it should do exactly the same multiplications as the existing code.

Can the shift operator do something wrong on diferent environments?

The method Buffer.writeBigInt64LE returns only the 32 bit part of the buffer as the current code does.

[Admdebian]

Update:

Current pyth code:

var val = BigInt(buffer[offset + 4]) + BigInt(buffer[offset + 5] * Math.pow(2, 8)) + BigInt(buffer[offset + 6] * Math.pow(2, 16)) + (last << 24)

if val === 14, I see in logs that val << 32 === 14

Then:

val << 31 === 0
val << 33 === 28

If I force BigInt I get same results: BigInt(BigInt(val) << BigInt(32))

I think that the operator works only with 32 bit in react-native.

[Admdebian]

From the DOC:

**JavaScript Uses 32 bits Bitwise Operands**
JavaScript stores numbers as 64 bits floating point numbers, but all bitwise operations are performed on 32 bits binary numbers.

Before a bitwise operation is performed, JavaScript converts numbers to 32 bits signed integers.

After the bitwise operation is performed, the result is converted back to 64 bits JavaScript numbers.

https://www.w3schools.com/js/js_bitwise.asp

[jayantk]

Ok, interesting. So when I run:

let val = 14;
console.log(BigInt(BigInt(val) << BigInt(32)));
console.log(val << 32);

I get 60129542144n for the first and 14 for the second. That makes sense to me because js only has 32-bit bitwise operators. You're saying the first one also gives you 14 in react-native? If so, it seems like BigInt isn't working properly (?) This seems surprising to me, but it could be true. I'm trying to google and see if i can find anyone else who has encountered the same problem, but I can't seem to find anything.

[Admdebian]

Ok, interesting. So when I run:

let val = 14;
console.log(BigInt(BigInt(val) << BigInt(32)));
console.log(val << 32);

I get 60129542144n for the first and 14 for the second. That makes sense to me because js only has 32-bit bitwise operators. You're saying the first one also gives you 14 in react-native? If so, it seems like BigInt isn't working properly (?) This seems surprising to me, but it could be true. I'm trying to google and see if i can find anyone else who has encountered the same problem, but I can't seem to find anything.

LOG BigInt(BigInt(val) << BigInt(32)): "14"
LOG val << 32: 14

Without shim.js and additional modules, Buffer and BigInt are not supported in react-native. We are surprised like you.

If this problem has happened to us, it can happen to other projects.

[jayantk]

Oh, I think i know what the problem is. Can you try BigInt(val).shiftLeft(32) and see if that works? I think the issue is that the big-integer library you are loading doesn't work with the javascript math operators, and instead requires you to use methods on the class.

[Admdebian]

Oh, I think i know what the problem is. Can you try BigInt(val).shiftLeft(32) and see if that works? I think the issue is that the big-integer library you are loading doesn't work with the javascript math operators, and instead requires you to use methods on the class.

Output:

LOG  BigInt(val).shiftLeft(32): "60129542144"
LOG  val << 32: 14

BigInt.shiftLeft is not defined without a new package. I installed:

"BigInt": "^5.5.3"

Then I imported:

import BigInt from 'big-integer'

Problem:

* output type bigint is not compatible with BigInteger from new module [email protected].
* BigInteger.valueOf returns a number.

[jayantk]

I think you want to add the module https://www.npmjs.com/package/big-integer , not BigInt .

[jayantk]

@Admdebian sorry for the late reply -- went on vacation and totally forgot about this for a few days. Hopefully the last comment makes sense to you and resolves the issue. lmk if not.

[Admdebian]

I think you want to add the module https://www.npmjs.com/package/big-integer , not BigInt .

Same problem. You can try with the last push. It doesn't convert BigInt to bigint.

I think if we change bigint to BigInt we introduce a breaking change.

In this project they moved from bigint to BigInt: https://github.com/prisma/prisma/pull/5842/files

[jayantk]

Ok, I consulted with some of our javascript gurus, and they said that (1) there is a native BigInt type built-in to javascript that is a wrapper around bigint, and (2) returning that BigInt is probably the right thing to do. I think that means you can remove the big-integer library entirely, then change the return type of the methods to BigInt, and that should work (?)

(the javascript big integer ecosystem is confusing. appreciate you bearing with me here.)

[ebramanti]

Hey @jayantk have you guys looked into using jsbi instead? I've noticed an issue with older Safari versions that do not support BigInt and there's no way to polyfill correctly because of the syntax.

Edit: Ah, see that this is addressed in #5