Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS

Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.

Use SQL to build ELT pipelines on a data lakehouse.

Lakehouse storage system benchmark

Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testing.

The open-source, AI-native data stack

Jupyter notebooks and AWS CloudFormation template to show how Hudi, Iceberg, and Delta Lake work

An open-source, community-driven REST catalog for Apache Iceberg!

Stream CDC into an Amazon S3 data lake in Apache Iceberg table format with AWS Glue Streaming and DMS

Sample code to collect Apache Iceberg metrics for table monitoring

Streaming ETL job cases in AWS Glue to integrate Iceberg and creating an in-place updatable data lake on Amazon S3

A sample implementation of stream writes to an Iceberg table on GCS using Flink and reading it using Trino

This repo contains examples of high throughput ingestion using Apache Spark and Apache Iceberg. These examples cover IoT and CDC scenarios using best practices. The code can be deployed into any Spark compatible engine like Amazon EMR Serverless or AWS Glue. A fully local developer environment is also provided.

Hands-on workshop with Iceberg, Redpanda, Debezium and Kafka-Connect

Hands-on workshop with Apache Iceberg

Miscellaneous codes and writings for MLOps

This is a collecton of Amazon CDK projects to show how to directly ingest streaming data from Amazon Mananged Service for Apache Kafka (MSK) and MSK Serverless into Apache Iceberg table in S3 with AWS Glue Streaming.

Using Apache Flink to write to s3 in Apache Iceberg format

Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming using Amazon MSK Serverless and MSK Connect (Debezium)

Run an open-source data LakeHouse locally using Docker Compose