-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
all: implement Spanner connector #59
Commits on Sep 23, 2023
-
ScanBuilder: implement buildScan to construct SQL
Allows us to build SQL from filters and required columns. Updates #58
Configuration menu - View commit details
-
Copy full SHA for a363750 - Browse repository at this point
Copy the full SHA a363750View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8489767 - Browse repository at this point
Copy the full SHA 8489767View commit details -
Configuration menu - View commit details
-
Copy full SHA for e68e4f8 - Browse repository at this point
Copy the full SHA e68e4f8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 197916c - Browse repository at this point
Copy the full SHA 197916cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7ec5d50 - Browse repository at this point
Copy the full SHA 7ec5d50View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9baf076 - Browse repository at this point
Copy the full SHA 9baf076View commit details -
Configuration menu - View commit details
-
Copy full SHA for b50f924 - Browse repository at this point
Copy the full SHA b50f924View commit details -
Configuration menu - View commit details
-
Copy full SHA for da84649 - Browse repository at this point
Copy the full SHA da84649View commit details -
Configuration menu - View commit details
-
Copy full SHA for c529dc3 - Browse repository at this point
Copy the full SHA c529dc3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5a2c260 - Browse repository at this point
Copy the full SHA 5a2c260View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8556acf - Browse repository at this point
Copy the full SHA 8556acfView commit details -
Configuration menu - View commit details
-
Copy full SHA for cfbb77e - Browse repository at this point
Copy the full SHA cfbb77eView commit details -
Add Map->JSON serializer and deserializer
Given that we've gotten crashes that Map<String, String> is non-serializable, this change adds a serializer and deserializer to JSON string.
Configuration menu - View commit details
-
Copy full SHA for bc8d205 - Browse repository at this point
Copy the full SHA bc8d205View commit details -
Configuration menu - View commit details
-
Copy full SHA for cc03ef4 - Browse repository at this point
Copy the full SHA cc03ef4View commit details -
Configuration menu - View commit details
-
Copy full SHA for ac9aef3 - Browse repository at this point
Copy the full SHA ac9aef3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0878137 - Browse repository at this point
Copy the full SHA 0878137View commit details -
Configuration menu - View commit details
-
Copy full SHA for ccdd23c - Browse repository at this point
Copy the full SHA ccdd23cView commit details -
Configuration menu - View commit details
-
Copy full SHA for f6f17b4 - Browse repository at this point
Copy the full SHA f6f17b4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 68a7dc1 - Browse repository at this point
Copy the full SHA 68a7dc1View commit details -
Did following steps to make the SpannerInputPartitionContext work wit…
…h serializable requirement. 1. Move the spanner generation inside createPartitionReaderContext to avoid making it serializable. 2. Remove the Closeable interface since spanner varible is not a member of the class. 3. Catch JsonProcessingException since Exception is too broad.
Configuration menu - View commit details
-
Copy full SHA for 2fd2591 - Browse repository at this point
Copy the full SHA 2fd2591View commit details -
Configuration menu - View commit details
-
Copy full SHA for e2b0c59 - Browse repository at this point
Copy the full SHA e2b0c59View commit details -
Configuration menu - View commit details
-
Copy full SHA for e80ccee - Browse repository at this point
Copy the full SHA e80cceeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6a13fbd - Browse repository at this point
Copy the full SHA 6a13fbdView commit details -
Configuration menu - View commit details
-
Copy full SHA for e8b0599 - Browse repository at this point
Copy the full SHA e8b0599View commit details -
Change the logger class to SpannerTable iteself. Changed the JSON typ…
…e to String. JSON inside spanner is not easy to convert to array.
Configuration menu - View commit details
-
Copy full SHA for cb5d733 - Browse repository at this point
Copy the full SHA cb5d733View commit details -
Making the following type conversion works from end to end: JSON, Byt…
…e, Date, and Timestamp.
Configuration menu - View commit details
-
Copy full SHA for 04f939e - Browse repository at this point
Copy the full SHA 04f939eView commit details -
Making the following type conversion works from end to end: JSON, Byt…
…e, Date, Timestamp and Array<String>.
Configuration menu - View commit details
-
Copy full SHA for 46f245c - Browse repository at this point
Copy the full SHA 46f245cView commit details -
Fixed an initialization issue. Removed the close() function in the Re…
…aderContext since spanner is not defined as a private member of the class.
Configuration menu - View commit details
-
Copy full SHA for cb80f6c - Browse repository at this point
Copy the full SHA cb80f6cView commit details -
Configuration menu - View commit details
-
Copy full SHA for ea7fdc8 - Browse repository at this point
Copy the full SHA ea7fdc8View commit details -
Configuration menu - View commit details
-
Copy full SHA for ffcdcaa - Browse repository at this point
Copy the full SHA ffcdcaaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6b7dd68 - Browse repository at this point
Copy the full SHA 6b7dd68View commit details -
Configuration menu - View commit details
-
Copy full SHA for c1f6e82 - Browse repository at this point
Copy the full SHA c1f6e82View commit details -
Configuration menu - View commit details
-
Copy full SHA for 81a75de - Browse repository at this point
Copy the full SHA 81a75deView commit details -
Use com.google.cloud.NoCredentials with SpannerEmulator+hermetic tests
Allows tests to run with the Cloud Spanner emulator without having to have GCP credentials present.
Configuration menu - View commit details
-
Copy full SHA for fd537f3 - Browse repository at this point
Copy the full SHA fd537f3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9417a17 - Browse repository at this point
Copy the full SHA 9417a17View commit details -
Try autoConfigEmulator=true;usePlainText with the right conditions
This allows the Cloud Spanner Emulator to be used in tests.
Configuration menu - View commit details
-
Copy full SHA for 2dcdfe5 - Browse repository at this point
Copy the full SHA 2dcdfe5View commit details -
Configuration menu - View commit details
-
Copy full SHA for d024244 - Browse repository at this point
Copy the full SHA d024244View commit details -
Configuration menu - View commit details
-
Copy full SHA for 04d5b3d - Browse repository at this point
Copy the full SHA 04d5b3dView commit details -
Fixed the cloudbuild.yaml cannot unmarshal string into Go value of ty…
…pe map[string]json.RawMessage
Configuration menu - View commit details
-
Copy full SHA for 2d1a57b - Browse repository at this point
Copy the full SHA 2d1a57bView commit details -
Configuration menu - View commit details
-
Copy full SHA for ac734b6 - Browse repository at this point
Copy the full SHA ac734b6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 00ec28f - Browse repository at this point
Copy the full SHA 00ec28fView commit details -
Configuration menu - View commit details
-
Copy full SHA for c5e890b - Browse repository at this point
Copy the full SHA c5e890bView commit details -
Added the gcp-setting.xml back to resolve the issue TransferFailedExc…
…eption: transfer failed for https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
Configuration menu - View commit details
-
Copy full SHA for e744e29 - Browse repository at this point
Copy the full SHA e744e29View commit details -
Fixed the issue that SpannerScanBuilderTest.testReadSchema:75 expecte…
…d:<6> but was:<0>
Configuration menu - View commit details
-
Copy full SHA for aafba1f - Browse repository at this point
Copy the full SHA aafba1fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 777bf67 - Browse repository at this point
Copy the full SHA 777bf67View commit details -
Configuration menu - View commit details
-
Copy full SHA for 841441e - Browse repository at this point
Copy the full SHA 841441eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 753bbde - Browse repository at this point
Copy the full SHA 753bbdeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 872dae0 - Browse repository at this point
Copy the full SHA 872dae0View commit details -
Configuration menu - View commit details
-
Copy full SHA for dc62d35 - Browse repository at this point
Copy the full SHA dc62d35View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9aeff03 - Browse repository at this point
Copy the full SHA 9aeff03View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6388369 - Browse repository at this point
Copy the full SHA 6388369View commit details -
Configuration menu - View commit details
-
Copy full SHA for c713512 - Browse repository at this point
Copy the full SHA c713512View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e0837b - Browse repository at this point
Copy the full SHA 9e0837bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9c4bc20 - Browse repository at this point
Copy the full SHA 9c4bc20View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1dd99ef - Browse repository at this point
Copy the full SHA 1dd99efView commit details -
Configuration menu - View commit details
-
Copy full SHA for 806b9ee - Browse repository at this point
Copy the full SHA 806b9eeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1e41554 - Browse repository at this point
Copy the full SHA 1e41554View commit details -
Configuration menu - View commit details
-
Copy full SHA for a25bbd1 - Browse repository at this point
Copy the full SHA a25bbd1View commit details -
Configuration menu - View commit details
-
Copy full SHA for bb540be - Browse repository at this point
Copy the full SHA bb540beView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4b2233f - Browse repository at this point
Copy the full SHA 4b2233fView commit details -
Configuration menu - View commit details
-
Copy full SHA for fdb031f - Browse repository at this point
Copy the full SHA fdb031fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7104076 - Browse repository at this point
Copy the full SHA 7104076View commit details -
Configuration menu - View commit details
-
Copy full SHA for d59a56a - Browse repository at this point
Copy the full SHA d59a56aView commit details -
Configuration menu - View commit details
-
Copy full SHA for ff8cc8c - Browse repository at this point
Copy the full SHA ff8cc8cView commit details -
Remove unnecessary spark.Batch from SpannerScanBuilder
Keep that functionality in SpannerScanner. While here also add filters to SpannerScanner to allow filtration.
Configuration menu - View commit details
-
Copy full SHA for 8241919 - Browse repository at this point
Copy the full SHA 8241919View commit details -
SpannerScanBuilder: implement SupportsPushDownRequiredColumns
This change implements a columns selector to reduce the amount of data returned and the results were confirmed by a live query ```python df.printSchema() df.select("created_at", "value", "base_cur") .filter((df["value"] > 3720) & (df["base_cur"] == "USD")) .show() ``` which produced ```shell root |-- id: string (nullable = false) |-- base_cur: string (nullable = false) |-- end_cur: string (nullable = false) |-- value: double (nullable = false) |-- data_src: string (nullable = false) |-- created_at: timestamp (nullable = false) |-- published_at: timestamp (nullable = true) +--------------------+-----------+--------+ | created_at| value|base_cur| +--------------------+-----------+--------+ |2023-08-30 20:56:...|3724.365901| USD| |2023-09-09 04:40:...|3738.384066| USD| |2023-09-07 05:44:...|3735.182945| USD| |2023-08-25 17:00:...|3730.606064| USD| |2023-09-15 10:08:...|3724.849835| USD| |2023-08-21 18:24:...|3727.353643| USD| |2023-08-20 07:52:...|3724.922281| USD| |2023-08-23 23:00:...|3738.342818| USD| |2023-08-14 10:00:...|3749.635375| USD| |2023-09-01 05:48:...|3730.472718| USD| |2023-09-15 21:40:...|3730.465981| USD| |2023-09-11 16:40:...|3723.400239| USD| |2023-09-13 13:16:...|3723.537745| USD| |2023-08-19 14:00:...|3724.922281| USD| |2023-09-06 02:56:...|3735.927727| USD| |2023-09-11 13:04:...|3723.400239| USD| |2023-09-14 17:56:...|3724.849835| USD| |2023-09-07 20:44:...|3739.570874| USD| |2023-08-22 07:00:...|3729.171921| USD| |2023-08-22 08:20:...|3729.171921| USD| +--------------------+-----------+--------+ ```
Configuration menu - View commit details
-
Copy full SHA for ae94dcc - Browse repository at this point
Copy the full SHA ae94dccView commit details -
SpannerUtils: add UserAgent when creating batchClient
Helps identify the connector to Google Cloud so that later usage metrics, health checks, quota updates, optimizations can be trivially made.
Configuration menu - View commit details
-
Copy full SHA for 8fe090f - Browse repository at this point
Copy the full SHA 8fe090fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 06ca6f8 - Browse repository at this point
Copy the full SHA 06ca6f8View commit details -
SpannerScanner: add option to disableDataboost
Allows Databoost to be disabled; it is on by default given the point of this connector. However, there is something to be said about compatibility so that by default most users who haven't enabled Databoost can still use it, but that's to be discussed for later. Fixes #68
Configuration menu - View commit details
-
Copy full SHA for cd66665 - Browse repository at this point
Copy the full SHA cd66665View commit details -
Configuration menu - View commit details
-
Copy full SHA for dd19c67 - Browse repository at this point
Copy the full SHA dd19c67View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6c241a3 - Browse repository at this point
Copy the full SHA 6c241a3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 10ec27c - Browse repository at this point
Copy the full SHA 10ec27cView commit details -
test: introduce SparkFilterUtilsTest and retrofit for that code
While here also fixed up the package path to be fully: package com.google.cloud.spark.spanner; instead of erroneously: package com.google.cloud.spark;
Configuration menu - View commit details
-
Copy full SHA for 5e3fcba - Browse repository at this point
Copy the full SHA 5e3fcbaView commit details -
make enableDataboost=true explicit and not an inversion with disableD…
…ataboost Hao reasoned that databoost being enabled should be an opt-in because it is expensive for customers so that's a good reason to require it to be explicitly specified by the customer.
Configuration menu - View commit details
-
Copy full SHA for 0a95456 - Browse repository at this point
Copy the full SHA 0a95456View commit details -
SpannerScanner: correctly close BatchClient in .planInputPartitions
This fixes a long standing shutdown failure due to unclosed Spanner objects.
Configuration menu - View commit details
-
Copy full SHA for a943057 - Browse repository at this point
Copy the full SHA a943057View commit details -
Configuration menu - View commit details
-
Copy full SHA for e5758d2 - Browse repository at this point
Copy the full SHA e5758d2View commit details -
Configuration menu - View commit details
-
Copy full SHA for cb58880 - Browse repository at this point
Copy the full SHA cb58880View commit details -
Configuration menu - View commit details
-
Copy full SHA for 47a95bd - Browse repository at this point
Copy the full SHA 47a95bdView commit details -
Configuration menu - View commit details
-
Copy full SHA for ec25270 - Browse repository at this point
Copy the full SHA ec25270View commit details -
Configuration menu - View commit details
-
Copy full SHA for 493eea2 - Browse repository at this point
Copy the full SHA 493eea2View commit details