Support for Array data-type

[amirbehzad]

Hi,
I noticed that this library does not support Athena's array data-type,
https://docs.aws.amazon.com/athena/latest/ug/data-types.html

Is there a plan to add this?

[tejasmanohar]

No plans personally, but PRs with tests will be accepted. Same for maps and structs.

[eatonphil]

I've got a project that makes use of struct, array, and map types. And I've been looking into adding support for them to this project. However dealing with these types the way scalar types are converted in https://github.com/segmentio/go-athena/blob/master/value.go#L32 seems not ideal. To proceed in that manner is to add string parsing functions (which I started to do) to support the various complex data types. This seems backwards because the data is provided structured until you ask for VarCharValue. Once we're acting on strings we've lost type information so you just have to guess about the types of nested data. Additionally I'm not even sure if it's possible to parse the strings from VarCharValue because (for example) strings in structs, maps, and arrays don't seem to be escaped with quotes? So I've seen array string values like this [foo, bar, biz] where foo, bar, biz are all strings. But what happens if one of those strings contains a comma?

I'm not sure VarCharValue is intended to be used for parsing (maybe I'm wrong though) and in any case it seems like this package is currently overcomplicating things by asking for the string value of structured data and then parsing the string value back into structured data as is done in convertValue. Furthermore, there is another function on the Datum, String, that calls a prettify function in awsutil that uses reflection to pull out the structured data.

All of this leads me to say that I think convertValue should be rewritten to use reflection to be most safe and support complex data sanely. But I wanted to post this here to make sure I'm understanding correctly and 1) if there's work already being done on this or 2) if I provide this work will it be upstreamed?

Thank you!

[eatonphil]

Here is a diff with the string-parsing route I was beginning to take. As you can see it gets hairy.

Additionally it is not JSON formatted as the outstanding PR suggests. It has its own string format (or one I don't recognize) that doesn't appear to be parseable when strings in arrays/maps/structs contain spaces or commas like I mentioned.

[eatonphil]

On further inspection I don't believe I'm right about the reflection pass being any different than what is happening here already. But I'm still confused how to retrieve structured data when string values are not quoted.

I filed an issue on the aws go sdk. aws/aws-sdk-go#2686