Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDFS/Downloads Table Incremental Build #1078

Open
fmendezh opened this issue Jun 27, 2024 · 0 comments
Open

HDFS/Downloads Table Incremental Build #1078

fmendezh opened this issue Jun 27, 2024 · 0 comments
Assignees
@muttcg
Labels
research research, evaluation or proof of concept

Comments

@fmendezh
Copy link
Contributor

The TableBackfill supports the creating of partitioned tables (by datasetKey), Apache Iceberg partitions supports the atomic update/replace of table partitions INSERT OVERWRITE TABLE occurrence PARTITION (dataset_key).
This feature can be used to keep the download table up-to-date and use the current table build process for disaster recovery and data synchronisation.
@fmendezh fmendezh added the research research, evaluation or proof of concept label Jun 27, 2024
fmendezh added a commit to gbif/occurrence that referenced this issue Aug 16, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
c01a9bc
fmendezh added a commit that referenced this issue Sep 12, 2024
@fmendezh
#1078 keeping table files
cc6bb0e
fmendezh added a commit that referenced this issue Sep 12, 2024
@fmendezh
#1078 changing version and keeping table files
d7e1d9d
fmendezh added a commit that referenced this issue Sep 12, 2024
@fmendezh
#1078 asserting the table directories exists after running the pipeline
1f8b957
fmendezh added a commit that referenced this issue Sep 12, 2024
@fmendezh
#1078 asserting the table directories exists after running the pipeline
be2309e
fmendezh added a commit that referenced this issue Sep 12, 2024
@fmendezh
#1078 asserting the table directories exists after running the pipeline
efd9a52
fmendezh added a commit that referenced this issue Sep 13, 2024
@fmendezh
#1078 formatting
c2566db
fmendezh added a commit that referenced this issue Sep 13, 2024
@fmendezh
#1078 fixing IT test, some table outputs must be absent when the inte…
ecfb06a 
…rpretation for that table is not requested
fmendezh added a commit that referenced this issue Sep 13, 2024
@fmendezh
#1078 formatting
ac238be
fmendezh added a commit that referenced this issue Sep 20, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
92b2970 
adding CLI components for incremental table build using iceberg
fmendezh added a commit that referenced this issue Sep 23, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
1f9acb0 
routing was using the wrong message
fmendezh added a commit that referenced this issue Sep 23, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
a6ca112 
adding balancer handler
fmendezh added a commit to gbif/gbif-api that referenced this issue Sep 24, 2024
@fmendezh
Data warehouse step
4ca6a9a 
gbif/pipelines#1078
fmendezh added a commit that referenced this issue Sep 24, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
4cfa71f 
changing step type for the data warehouse cli
fmendezh added a commit that referenced this issue Sep 24, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
400d69f 
changing step type for the data warehouse cli
fmendezh added a commit that referenced this issue Sep 24, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
391ada1 
reverting issue count
fmendezh added a commit that referenced this issue Sep 24, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
b81871a 
fixing it test, steps count
fmendezh added a commit that referenced this issue Sep 24, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
5bf7233 
fixing it test, steps count
fmendezh added a commit to gbif/registry that referenced this issue Sep 25, 2024
@fmendezh
Data warehouse step
642746f 
gbif/pipelines#1078
fmendezh added a commit to gbif/registry that referenced this issue Sep 25, 2024
@fmendezh
changing artifact version
6d0b27d 
gbif/pipelines#1078
fmendezh added a commit to gbif/postal-service that referenced this issue Sep 26, 2024
@fmendezh
adding warehouse message
f8fce47 
gbif/pipelines#1078
fmendezh added a commit that referenced this issue Sep 26, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
f464cb0 
moving data warehouse message to postal service
fmendezh added a commit to gbif/registry that referenced this issue Sep 26, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
c6b1fed 
adding data warehouse message processing
fmendezh added a commit to gbif/postal-service that referenced this issue Sep 26, 2024
@fmendezh
adding warehouse message
7e2752e 
gbif/pipelines#1078
fmendezh added a commit to gbif/postal-service that referenced this issue Sep 26, 2024
@fmendezh
https://github.com/gbif/pipelines/issues/1078
08879c4 
using latest snapshot
fmendezh added a commit that referenced this issue Oct 11, 2024
@fmendezh
#1078 keeping table files
6f1eab0
fmendezh added a commit that referenced this issue Oct 11, 2024
@fmendezh
Revert "#1078 keeping table files"
3f976aa 
This reverts commit 6f1eab0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@muttcg muttcg

Labels
research research, evaluation or proof of concept
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fmendezh @muttcg