You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I think very large JSON files are not ideal for many reasons, but that being said there may be methods we could implement to accommodate large JSON files if we need to.
The 512MB string size limit is a hard limit in Node.js and when our processor reads the JSON file it accumulates all the chunks into a string to then parse to JSON, but with large files I get this error:
"error": "TSError: Cannot create a string longer than 0x1fffffe8 characters\n
I think very large JSON files are not ideal for many reasons, but that being said there may be methods we could implement to accommodate large JSON files if we need to.
The 512MB string size limit is a hard limit in Node.js and when our processor reads the JSON file it accumulates all the chunks into a string to then parse to JSON, but with large files I get this error:
We could try something like - https://www.npmjs.com/package/stream-json to stream process json files, or there are other streaming options.
This tickets lists some more options - https://stackoverflow.com/questions/68230031/cannot-create-a-string-longer-than-0x1fffffe8-characters-in-json-parse.
At the very least we should add a better error explanation.
The text was updated successfully, but these errors were encountered: