spannerspark: initial code skeleton for read #31
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/cc @halio-g |
odeke-em
force-pushed
the
initial-code-wireup
branch
from
7ed70df
to
7991c4f
Compare
halio-g
reviewed
Jul 21, 2023
examples/SpannerSpark.java
Outdated
| Dataset<Row> df = spark.read() | ||
| .format("cloud-spanner") | ||
| .option("table", "people") | ||
| .option("projectId", "orijtech-161805") |
There was a problem hiding this comment.
Please leave the personal or test information empty. You can keep it in your local branch though.
There was a problem hiding this comment.
Yeah I am eventually going to read it from System.getenv
halio-g
reviewed
Jul 21, 2023
| import java.util.concurrent.TimeUnit; | ||
|
|
||
| public class SpannerSpark implements DataSourceRegister, TableProvider { | ||
| public Dataset<Row> execute(SparkSession spark, String sqlStmt) { |
There was a problem hiding this comment.
Let's hide the code that is unused by the inherited method.
halio-g
reviewed
Jul 21, 2023
| // "projectId": <PROJECT_ID> | ||
| // "databaseId": <DATABASE_ID> | ||
| String spannerUri = String.format( | ||
| "cloudspanner:/projects/%s/instances/%s/databases/%s", |
There was a problem hiding this comment.
Probably we want to extract the usage of connectionOptions and Spanner operations into multiple class for easy testing.
halio-g
reviewed
Jul 21, 2023
spark-3.1-spanner-lib/src/main/java/com/google/cloud/spark/spanner/SpannerSpark.java
Outdated
Show resolved
Hide resolved
halio-g
reviewed
Jul 21, 2023
spark-3.1-spanner-lib/src/main/java/com/google/cloud/spark/spanner/SpannerSpark.java
Outdated
Show resolved
Hide resolved
| import java.util.concurrent.Executors; | ||
| import java.util.concurrent.TimeUnit; | ||
|
|
||
| public class SpannerSpark implements DataSourceRegister, TableProvider { |
There was a problem hiding this comment.
Please call it SpannerTableProvider or so.
odeke-em
force-pushed
the
initial-code-wireup
branch
2 times, most recently
from
4e3a0d4
to
d308ed6
Compare
First phase of the code skeleton that should allow .printSchema() and .show() to work correctly as well as running queries.
odeke-em
force-pushed
the
initial-code-wireup
branch
2 times, most recently
from
3e0f0ac
to
531b211
Compare
odeke-em
force-pushed
the
initial-code-wireup
branch
4 times, most recently
from
d7a9368
to
7325137
Compare
odeke-em
force-pushed
the
initial-code-wireup
branch
from
7325137
to
74427e1
Compare
odeke-em
force-pushed
the
initial-code-wireup
branch
from
88dd79c
to
7b7122c
Compare
odeke-em
force-pushed
the
initial-code-wireup
branch
7 times, most recently
from
47456df
to
65729a9
Compare
halio-g
reviewed
Aug 1, 2023
halio-g
reviewed
Aug 1, 2023
spark-3.1-spanner-lib/src/main/java/com/google/cloud/spark/spanner/DefaultSource.java
Outdated
Show resolved
Hide resolved
halio-g
reviewed
Aug 1, 2023
spark-3.1-spanner-lib/src/main/java/com/google/cloud/spark/spanner/SpannerDataReader_java
Outdated
Show resolved
Hide resolved
halio-g
reviewed
Aug 1, 2023
spark-3.1-spanner-lib/src/main/java/com/google/cloud/spark/spanner/SpannerDataSourceReader_java
Outdated
Show resolved
Hide resolved
halio-g
reviewed
Aug 1, 2023
spark-3.1-spanner-lib/src/test/java/com/google/cloud/spark/spanner/SpannerSparkTest.java
Outdated
Show resolved
Hide resolved
halio-g
reviewed
Aug 1, 2023
spark-3.1-spanner-lib/src/test/java/com/google/cloud/spark/spanner/SpannerSparkTest.java
Outdated
Show resolved
Hide resolved
halio-g
reviewed
Aug 1, 2023
spark-3.1-spanner-lib/src/test/java/com/google/cloud/spark/spanner/SpannerSparkTest.java
Outdated
Show resolved
Hide resolved
halio-g
reviewed
Aug 1, 2023
spark-3.1-spanner-lib/src/test/java/com/google/cloud/spark/spanner/SpannerSparkTest.java
Outdated
Show resolved
Hide resolved
halio-g
reviewed
Aug 1, 2023
| this.properties = properties; | ||
| } | ||
|
|
||
| public Dataset<Row> execute(SparkSession spark, String sqlStmt) { |
There was a problem hiding this comment.
This function is not used anywhere. Let's remove it in the pull request to make the pr clean.
Fixes #46
This change implements SpannerPartitionReader which takes in an InputPartition which would have been generated after running a partitionQuery that then serializes the returned Partition over the network to the executor which when creating SpannerPartitionReader acquires a transaction then retrieves values from the produced ResultSet. Later on, we shall need to investigate if the transaction might be held for longer than 10 seconds and might be idle for that long.
|
I am going to merge the base working version in and then individually work on the respective issues plus split work. |
This was referenced Aug 14, 2023
Closed
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First phase of the code skeleton that should allow .printSchema() and .show() to work correctly as well as running queries.