deploying dbt docs

Docker/Dockerfile.connext-dbt-docs

@@ -1,9 +1,25 @@
-# Dockerfile: Dockerfile.dbt-docs
-FROM fishtownanalytics/dbt:0.21.0
+FROM python:3.11.4
+
+ENV LANG C.UTF-8
+ENV LC_ALL C.UTF-8
+ENV PYTHONFAULTHANDLER 1
 
 WORKDIR /app
-COPY connext_dbt /app
-RUN dbt docs generate
+COPY Pipfile.lock Pipfile ./
+
+# Building dependencies
+RUN pip install pipenv
+RUN pipenv install --system --deploy
+
+# Copy the shell script into the Docker image
+COPY generate_dbt_docs.sh ./generate_dbt_docs.sh
+
+# Make the shell script executable
+RUN chmod +x ./generate_dbt_docs.sh
+
+COPY . .
 
 EXPOSE 8080
-CMD ["dbt", "docs", "serve", "--port", "9000", "--no-browser"]
+
+# Use the shell script as the Docker CMD
+CMD [ "./generate_dbt_docs.sh" ]

cloudbuilds/cloudbuild-service-connext-dbt-docs.yaml

@@ -0,0 +1,13 @@
+steps:
+  - name: "gcr.io/cloud-builders/docker"
+    args:
+      [
+        "build",
+        "-t",
+        "us-central1-docker.pkg.dev/mainnet-bigq/dbt/connext-dbt-docs",
+        "-f",
+        "Docker/Dockerfile.connext-dbt-docs",
+        "."
+      ]
+images:
+  - "us-central1-docker.pkg.dev/mainnet-bigq/dbt/connext-dbt-docs"

connext_dbt/models/bridges/bridges.md

@@ -0,0 +1,123 @@
+# Bridges Models Documentation
+
+## Overview
+
+The Bridges section of the Connext DBT project focuses on transforming and modeling data related to various blockchain bridges. This includes staging raw transaction data, cleaning and formatting it, and creating mart models optimized for reporting and analytics.
+
+## Models
+
+### `all_combined_bridges_txs`
+
+- **ID Column:**
+  - `id`
+
+- **Tests:**
+  - `not_null`
+  - `unique`
+
+- **Overview:**
+  Combines data from multiple bridge transaction sources (`cln_hop_txs`, `cln_debridge_txs`, `cln_symbiosis_txs`, `cln_synapse_txs`, `cln_all_bridge_txs`) to create a unified view of all bridge transactions. This model ensures that all records from different bridges are included using the `UNION ALL` operator.
+
+- **Transformations:**
+  Aggregates and unifies transaction data from various source tables by performing a `UNION ALL`. This ensures a comprehensive dataset that includes all bridge transactions across different protocols.
+
+### `all_combined_bridges_txs_modelling_format`
+
+- **ID Column:**
+  - `id`
+
+- **Tests:**
+  - None specified.
+
+- **Overview:**
+  Formats the combined bridge transaction data for analysis. It includes fields such as timestamps, amounts, fees, user addresses, and chain information, ensuring that the data is ready for downstream analytics.
+
+- **Transformations:**
+  Maps and renames columns to standardize field names across different bridge sources. Converts data types and structures timestamps to ensure consistency, facilitating easier analysis and reporting.
+
+### `cln_debridge_txs`
+
+- **ID Column:**
+  - `id`
+
+- **Tests:**
+  - `not_null`
+  - `unique`
+
+- **Overview:**
+  Cleans and transforms data from the `stg_debridge_txs` table by renaming columns and converting data types. This preparation ensures data consistency and readiness for further analysis.
+
+- **Transformations:**
+  Renames columns to maintain uniform naming conventions and casts data types to appropriate formats. Filters and cleans records to remove any inconsistencies or irrelevant data points.
+
+### `cln_hop_txs`
+
+- **ID Column:**
+  - `id`
+
+- **Tests:**
+  - `not_null`
+  - `unique`
+
+- **Overview:**
+  Cleans and transforms data from the `stg_hop_txs` table. It renames columns and converts data types to ensure the data is correctly formatted for analysis.
+
+- **Transformations:**
+  Standardizes column names and data types. Applies necessary formatting to prepare the transaction data for seamless integration with other bridge data sources.
+
+### `cln_symbiosis_txs`
+
+- **ID Column:**
+  - `id`
+
+- **Tests:**
+  - `not_null`
+  - `unique`
+
+- **Overview:**
+  Extracts and cleans transfer records from the `source_symbiosis_bridge_explorer_transactions` table for the staging layer. It converts timestamps and filters records to ensure data relevancy and accuracy.
+
+- **Transformations:**
+  Converts raw timestamps to standardized formats and filters transactions based on specific criteria to ensure only relevant and accurate data is retained for analysis.
+
+### `cln_synapse_txs`
+
+- **ID Column:**
+  - `id`
+
+- **Tests:**
+  - `not_null`
+  - `unique`
+
+- **Overview:**
+  Cleans and transforms data from the `stg_synapse_txs` table by renaming columns and converting data types. Prepares the data for further analysis by ensuring all necessary fields are correctly formatted.
+
+- **Transformations:**
+  Standardizes column naming and data types. Ensures consistency across Synapse transactions, facilitating their integration with other bridge transaction data.
+
+## Mart Models Lineage
+
+### `all_combined_bridges_txs`
+
+- **Source Models:**
+  - `cln_hop_txs`
+  - `cln_debridge_txs`
+  - `cln_symbiosis_txs`
+  - `cln_synapse_txs`
+  - `cln_all_bridge_txs`
+
+- **Description:**
+  Aggregates transaction data from the above source models to provide a comprehensive view of all bridge transactions within the data warehouse.
+
+### `all_combined_bridges_txs_modelling_format`
+
+- **Source Model:**
+  - `all_combined_bridges_txs`
+
+- **Description:**
+  Formats and structures the aggregated bridge transactions for analytical purposes, adding necessary fields and ensuring data integrity for reporting.
+
+## Summary
+
+The Bridges models within the Connext DBT project are crucial for consolidating and preparing bridge transaction data from various sources. By staging, cleaning, and aggregating this data, the project ensures that stakeholders have access to reliable and comprehensive analytics on blockchain bridge activities.
+

connext_dbt/models/bridges/mart/all_combined_bridges_txs.sql

@@ -5,3 +5,5 @@ UNION ALL
 SELECT * FROM {{ ref('cln_symbiosis_txs') }}
 UNION ALL
 SELECT * FROM {{ ref('cln_synapse_txs') }}
+UNION ALL
+SELECT * FROM {{ ref('cln_all_bridge_txs') }}

connext_dbt/models/overview.md

@@ -0,0 +1,46 @@
+{% docs __overview__ %}
+# Everclear Data Warehouse
+
+**DOCS UPDATE DATE: 29th September 2024 | DOC VERSION: 0.0.1 | In Progress(Doc might not be up to date with the latest code/models)**
+
+Welcome to the Connext Analytics DBT Documentation. This landing page provides an overview of the project, its models, and useful resources.
+
+## Project Overview
+
+The Connext DBT project is designed to transform and model data related to blockchain bridges, liquidity metrics, and user interactions. The project leverages **dbt** to ensure data integrity, scalability, and maintainability.
+
+## Models
+
+### Metrics Models- TBD
+
+Metrics models compute key performance indicators and aggregate data for analysis.
+
+
+### Staging Models
+
+Staging models handle the initial extraction and cleansing of raw data from various sources.
+
+#### Bridges
+
+- **Bridges Staging Models**
+  - [`stg_all_bridge_txs`](models/bridges/stage/stg_all_bridge_txs.sql): Extracts and deduplicates bridge transaction data.
+  - [`stg_symbiosis_txs`](models/bridges/stage/stg_symbiosis_txs.sql): Processes Symbiosis bridge transactions.
+  - [`stg_connext_txs`](models/bridges/stage/stg_connext_txs.sql): Processes Connext bridge transactions.
+  - [`stg_hop_txs`](models/bridges/stage/stg_hop_txs.sql): Processes Hop bridge transactions.
+  - [`stg_debridge_txs`](models/bridges/stage/stg_debridge_txs.sql): Processes Debridge bridge transactions.
+  - [`stg_synapse_txs`](models/bridges/stage/stg_synapse_txs.sql): Processes Synapse bridge transactions.
+  - [`stg_all_bridge_txs`](models/bridges/stage/stg_all_bridge_txs.sql): Processes all bridge transactions
+
+#### Bridges Mart Models
+
+Mart models are optimized for reporting and analytics, combining data from multiple sources.
+
+  - [`all_combined_bridges_txs`](models/bridges/mart/all_combined_bridges_txs.sql): Merges transactions from multiple bridge sources.
+  - [`all_combined_bridges_txs_modelling_format`](models/bridges/mart/all_combined_bridges_txs_modelling_format.sql): Formats combined transaction data for analysis.
+
+## Resources
+
+- **Project Repository:** [GitHub](https://github.com/connext/connext-analytics)
+- **DBT Documentation:** [dbt Docs](https://github.com/connext/connext-analytics/tree/main/connext_dbt)
+
+{% enddocs %}