You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
I'm loading data output by yyjson into Spark/Hive tables, and the fact that empty objects are output as {} instead of null is causing a lot of issues, because Spark is not able to infer any meaningful schema and ends up dropping the empty object fields.
Describe the solution you'd like
Output empty objects as null instead of {}. I'd be happy to have to use a flag to achieve this.
Describe alternatives you've considered
The only alternatives I can see would be outside the scope of yyjson, eg:
post-processing to convert all {} to null - this feels like it would be really slow and defeat a lot of the point of using yyjson.
upstream code changes to ensure we define a schema for our empty dictionaries, so {} becomes {optional_field_1: null, optional_field_2:null, ...} - this feels "wrong", since there is a distinction between a null object and an object with keys and null values.
define a full schema for our Spark tables. This will work, but it's not how I'd like to do things on this particular project for other reasons.
Additional context
N/A
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
I'm loading data output by yyjson into Spark/Hive tables, and the fact that empty objects are output as
{}
instead ofnull
is causing a lot of issues, because Spark is not able to infer any meaningful schema and ends up dropping the empty object fields.Describe the solution you'd like
Output empty objects as
null
instead of{}
. I'd be happy to have to use a flag to achieve this.Describe alternatives you've considered
The only alternatives I can see would be outside the scope of yyjson, eg:
{}
tonull
- this feels like it would be really slow and defeat a lot of the point of using yyjson.{}
becomes{optional_field_1: null, optional_field_2:null, ...}
- this feels "wrong", since there is a distinction between a null object and an object with keys and null values.Additional context
N/A
The text was updated successfully, but these errors were encountered: