-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
added information into the README.md
- Loading branch information
Showing
2 changed files
with
32 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,32 @@ | ||
# aws_etl_pipeline | ||
# Architecture | ||
|
||
![Pipeline Architecture](images/architecture.png) | ||
|
||
# Tech Stack | ||
- Terraform | ||
- Github actions (CI/CD) | ||
- AWS Glue Data Catalog | ||
- AWS Glue Crawler | ||
- AWS Glue Trigger | ||
- AWS Glue Classifier | ||
- AWS Glue ETL Job | ||
- AWS Lambda | ||
- AWS Eventbridge | ||
- Amazon S3 | ||
- Amazon Athena | ||
- SQL | ||
- Python | ||
|
||
# Overwiew | ||
|
||
In this project, I have create an ETL Job on AWS using Terraform. | ||
The project extract data from an API (Zillow) which are data from real estate, then, process it using AWS ETL Glue Job with Spark. | ||
Data is extracted from the API using lambda function which is scheduled to run every day. At the end, the data is stored in an s3 bucket in a JSON format. | ||
|
||
The AWS Crawler then crawl the data and create a table in glue data catalog, then use AWS ETL job with Spark to process that real estate data and build a report to showcast, for each state, country the price per sqft. | ||
|
||
For more information you can check this meduim article : | ||
- [How I build an ETL pipeline with AWS Glue, Lambda and Terraform](https://medium.com/@lorenagongang/how-i-build-an-etl-pipeline-with-aws-glue-lambda-and-terraform-bbdf0788cc75) | ||
|
||
|
||
|
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.