Skip to content
This repository has been archived by the owner on Feb 26, 2024. It is now read-only.

Create a web3 http provider that can handle massive JSON blobs without crashing due to string or buffer length issues #4381

Draft
wants to merge 7 commits into
base: develop
Choose a base branch
from

Conversation

davidmurdoch
Copy link
Member

@davidmurdoch davidmurdoch commented Oct 29, 2021

Draft PR because I don't know the intricacies of Truffle well enough to be confident this works well enough. I'm not sure I replaced the right Web3.providers.HttpProvider calls.

This only supports http and https providers. It does NOT support ws(s) URLs. the default in truffle is http, so it may still be valuable to ship without a WebSocket version. I imagine WebSockets support will be a bit more complicated to patch in.

There is also only a single test added, and it's a strange one. The test mocks an RPC provider and just always returns a few gigs of data in the response. The test will fail with the infamous Invalid String Length error if the method in the RPC request isn't "debug_traceTransaction" (as expected). So it does seem to work. The test takes 15-20s to run on my machine because of the massive amount of data that needs to be parsed, which isn't ideal #helpwanted :-)

I've also smoke tested it with mainnet transaction 0x8bb8dc5c7c830bac85fa48acad2505e9300a91c3ff239c9517d0cae33b595090 (which generates a 1.4 GB trace) and it works.

I exported the class as default, but I don't have a good reason. It really should be exported as a named class.

Note: a ganache alpha that supports these giant traces will be released shortly after trufflesuite/ganache#1339 is merged.

result.structLogs.forEach((log: Buffer) => {
assert.strictEqual(log.length, SIZE);
});
}) as any).timeout(0);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what is up with the mocha types, but it thinks it() returns void but it doesn't.

* @param {Object} payload
* @param {Function} callback triggered on end with (err, result)
*/
send(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make this guy also implement request?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. This is a drop-in replacement for Web3HttpProvider, which doesn't implement request.

@cds-amal cds-amal self-assigned this Nov 3, 2021
@cds-amal cds-amal removed their assignment Jul 11, 2022
@davidmurdoch davidmurdoch force-pushed the fix/streaming-web3-http-provider branch from ae453d4 to 80339a1 Compare March 14, 2023 20:25
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants