Geoparquet filtering and simplification #895
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
bchapuis
commented
Oct 10, 2024
•
bchapuis
commented
- Adds bbox based filtering capabilities to the geoparquet reader, hence loading only relevant records to the db.
- Simplifies the object model to minimize the creation of wrappers and arrays to hold values.
- Removes unused classes and methods related to the geoparquet writer.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
baremaps-geoparquet/src/main/java/org/apache/baremaps/geoparquet/GeoParquetGroup.java
Fixed
Show fixed
Hide fixed
| } | ||
|
|
||
| private FileInfo getFileInfo(FileStatus fileStatus) { | ||
| try { | ||
| return buildFileInfo(fileStatus); | ||
| ParquetMetadata parquetMetadata = | ||
| ParquetFileReader.readFooter(configuration, fileStatus.getPath()); |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation Note
bchapuis
force-pushed
the
geoparquet-filtering-and-simplification
branch
from
07f53a3
to
930cc0f
Compare
| return getEnvelopeValues(parquetSchema.getFieldIndex(fieldName)); | ||
| } | ||
|
|
||
| public String toString() { |
Check notice
Code scanning / CodeQL
Missing Override annotation Note
bchapuis
force-pushed
the
geoparquet-filtering-and-simplification
branch
2 times, most recently
from
2703d41
to
b4a5f1a
Compare
bchapuis
force-pushed
the
geoparquet-filtering-and-simplification
branch
from
b4a5f1a
to
ffbc74c
Compare
bchapuis
force-pushed
the
geoparquet-filtering-and-simplification
branch
from
c602218
to
1f9cc17
Compare
...marking/src/main/java/org/apache/baremaps/benchmarking/geoparquet/OvertureMapsBenchmark.java
Fixed
Show fixed
Hide fixed
...marking/src/main/java/org/apache/baremaps/benchmarking/geoparquet/OvertureMapsBenchmark.java
Fixed
Show fixed
Hide fixed
...nchmarking/src/main/java/org/apache/baremaps/benchmarking/geoparquet/SmallFileBenchmark.java
Fixed
Show fixed
Hide fixed
...nchmarking/src/main/java/org/apache/baremaps/benchmarking/geoparquet/SmallFileBenchmark.java
Fixed
Show fixed
Hide fixed
|
@sebr72 I hope you are doing well. I refactored the GeoParquetGroup so that we don't need wrapper objects and instantiate arrays only when it is necessary. I also adapted the spliterator so that we can filter the files and the records based on their respective bbox. Please let me now what you think about these changes. |
bchapuis
force-pushed
the
geoparquet-filtering-and-simplification
branch
from
e00983f
to
5ea6c90
Compare
bchapuis
force-pushed
the
geoparquet-filtering-and-simplification
branch
from
5ea6c90
to
985ef2c
Compare
bchapuis
force-pushed
the
geoparquet-filtering-and-simplification
branch
from
42eacb7
to
cdb9f6a
Compare
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.