Merge pull request #2 from DP6/rebranding

Rebranding to Marketing Data Sync

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,27 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: '[BUG]'
+labels: bug
+assignees: ''
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,19 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: '[NEW]'
+labels: enhancement
+assignees: ''
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,5 @@
+**What issue does this pull request resolve?**
+
+**What changes did you make?**
+
+**Is there anything that requires more attention while reviewing?**

.github/dependabot.yml

@@ -0,0 +1,13 @@
+version: 2
+updates:
+  # Maintain dependencies for GitHub Actions
+  - package-ecosystem: 'github-actions'
+    directory: '/'
+    schedule:
+      interval: 'weekly'
+
+  # Maintain dependencies for npm
+  - package-ecosystem: 'pip'
+    directory: '/megalist_dataflow'
+    schedule:
+      interval: 'weekly'

.github/workflows/codacy-analysis.yml

@@ -0,0 +1,36 @@
+# This workflow checks out code, performs a Codacy security scan
+# and integrates the results with the
+# GitHub Advanced Security code scanning feature.  For more information on
+# the Codacy security scan action usage and parameters, see
+# https://github.com/codacy/codacy-analysis-cli-action.
+# For more information on Codacy Analysis CLI in general, see
+# https://github.com/codacy/codacy-analysis-cli.
+
+name: Codacy
+
+on: ['push']
+
+jobs:
+  codacy-security-scan:
+    name: Codacy Analysis
+    runs-on: ubuntu-latest
+    steps:
+      # Checkout the repository to the GitHub Actions runner
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      # Execute Codacy Analysis CLI and generate a SARIF output with the security issues identified during the analysis
+      - name: Run Codacy Analysis CLI
+        uses: codacy/[email protected]
+        with:
+          # Check https://github.com/codacy/codacy-analysis-cli#project-token to get your project token from your Codacy repository
+          # You can also omit the token and run the tools that support default configurations
+          project-token: ${{ secrets.CODACY_PROJECT_TOKEN }}
+          verbose: true
+          output: results.sarif
+          format: sarif
+          # Adjust severity of non-security issues
+          gh-code-scanning-compat: true
+          # Force 0 exit code to allow SARIF file generation
+          # This will handover control about PR rejection to the GitHub side
+          max-allowed-issues: 2147483647

CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to make participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+- Using welcoming and inclusive language
+- Being respectful of differing viewpoints and experiences
+- Gracefully accepting constructive criticism
+- Focusing on what is best for the community
+- Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+- The use of sexualized language or imagery and unwelcome sexual attention or
+  advances
+- Trolling, insulting/derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+- Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies within all project spaces, and it also applies when
+an individual is representing the project or its community in public spaces.
+Examples of representing a project or community include using an official
+project e-mail address, posting via an official social media account, or acting
+as an appointed representative at an online or offline event. Representation of
+a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at [email protected]. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

README.md

@@ -1,8 +1,22 @@
-# Megalista
+# MDS - Marketing Data Sync
 
-Sample integration code for onboarding offline/CRM data from BigQuery as custom audiences or offline conversions in Google Ads, Google Analytics 360, Google Display & Video 360 and Google Campaign Manager.
+Solution based on the [Google Megalista project](https://github.com/google/megalista).
 
-**Disclaimer:** This is not an officially supported Google product.
+<div align="center">
+  <img src="https://raw.githubusercontent.com/DP6/templates-centro-de-inovacoes/main/public/images/centro_de_inovacao_dp6.png" height="100px" />
+</div>
+
+<p align="center">
+  <a href="#badge">
+    <img alt="semantic-release" src="https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg">
+  </a>
+  <a href="https://www.codacy.com/gh/DP6/marketing-data-sync/dashboard?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=DP6/marketing-data-sync&amp;utm_campaign=Badge_Grade">
+    <img alt="Code quality" src="https://app.codacy.com/project/badge/Grade/4bb28565a8b241abae247e6e84778723"/>
+  </a>
+
+</p>
+
+Sample integration code for onboarding offline/CRM data from BigQuery as custom audiences or offline conversions in Google Ads, Google Analytics 360, Google Display & Video 360, Google Campaign Manager and Facebook Ads.
 
 ## Supported integrations
 - **Google Ads**
@@ -25,7 +39,7 @@ Sample integration code for onboarding offline/CRM data from BigQuery as custom
   - S2S Offline events API (conversion upload), to be used for audience creation and in-app events with Google Ads and DV360 [[details]](https://support.appsflyer.com/hc/en-us/articles/207034486-API-de-eventos-de-servidor-para-servidor-S2S-mobile-para-mobile)
 
 ## How does it work
-Megalista was design to separate the configuration of conversion/audience upload rules from the engine, giving more freedom for non-technical teams (i.e. Media and Business Inteligence) to setup multiple upload rules on their own.
+MDS was design to separate the configuration of conversion/audience upload rules from the engine, giving more freedom for non-technical teams (i.e. Media and Business Inteligence) to setup multiple upload rules on their own.
 
 The solution consists in #1 a Google Spreadsheet (template) in which all rules are defined by mapping a data source (BigQuery Table) to a destination (data upload endpoint) and #2, an apache beam workflow running on Google Dataflow, scheduled to upload the data in batch mode.
 
@@ -46,7 +60,7 @@ The solution consists in #1 a Google Spreadsheet (template) in which all rules a
 - **Google Cloud SDK**
 
 ### Access Requirements
-Those are the minimum roles necessary to deploy Megalista:
+Those are the minimum roles necessary to deploy MDS:
 - OAuth Config Editor
 - BigQuery User
 - BigQuery Job User
@@ -81,23 +95,23 @@ In order to create it, follow these steps:
  - On the **OAuth Consent Screen** and configure an *Application name*
  - Then, go to the **Credentials** and create an *OAuth client Id* with Application type set as *Desktop App*
  - This will generate a *Client Id* and a *Client secret*
- - Run the **generate_megalist_token.sh** script in this folder providing these two values and follow the instructions
-   - Sample: `./generate_megalist_token.sh client_id client_secret`
+ - Run the **generate_mds_token.sh** script in this folder providing these two values and follow the instructions
+   - Sample: `./generate_mds_token.sh client_id client_secret`
  - This will generate the *Access Token* and the *Refresh token*
 
 ### Creating a bucket on Cloud Storage
 This bucket will hold the deployed code for this solution. To create it, navigate to the *Storage* link on the top-left menu on GCP and click on *Create bucket*. You can use Regional location and Standard data type for this bucket.
 
-## Running Megalista
+## Running MDS
 
 We recommend first running it locally and make sure that everything works. 
 Make some sample tables on BigQuery for one of the uploaders and make sure that the data is getting correctly to the destination.
 After that is done, upload the Dataflow template to GCP and try running it manually via the UI to make sure it works.
-Lastly, configure the Cloud Scheduler to run Megalista in the frequency desired and you'll have a fully functional data integration pipeline.
+Lastly, configure the Cloud Scheduler to run MDS in the frequency desired and you'll have a fully functional data integration pipeline.
 
 ### Running locally
 ```bash
-python3 megalist_dataflow/main.py \
+python3 mds_dataflow/main.py \
   --runner DirectRunner \
   --developer_token ${GOOGLE_ADS_DEVELOPER_TOKEN} \
   --setup_sheet_id ${CONFIGURATION_SHEET_ID} \
@@ -119,15 +133,15 @@ To execute the pipeline, use the following steps:
 - Go to **Dataflow** on GCP console
 - Click on *Create job from template*
 - On the template selection dropdown, select *Custom template* 
-- Find the *megalist* file on the bucket you've created, on the templates folder
+- Find the *mds* file on the bucket you've created, on the templates folder
 - Fill in the parameters required and execute
 
 ### Scheduling pipeline
 To schedule daily/hourly runs, go to **Cloud Scheduler**:
 - Click on *create job* 
 - Add a name and frequency as desired
 - For *target* set as HTTP
-- Configure a *POST* for url: https://dataflow.googleapis.com/v1b3/projects/${YOUR_PROJECT_ID}/locations/${LOCATION}/templates:launch?gcsPath=gs://${BUCKET_NAME}/templates/megalist, replacing the params with the actual values
+- Configure a *POST* for url: https://dataflow.googleapis.com/v1b3/projects/${YOUR_PROJECT_ID}/locations/${LOCATION}/templates:launch?gcsPath=gs://${BUCKET_NAME}/templates/mds, replacing the params with the actual values
 - For a sample on the *body* of the request, check **cloud_config/scheduler.json**
 - Add OAuth Headers
 - Scope: https://www.googleapis.com/auth/cloud-platform
@@ -142,4 +156,18 @@ It's recommended to create a new Service Account to be used with the Cloud Sched
 
 
 ## Usage
-Every upload method expects as source a BigQuery data with specific fields, in addition to specific configuration metadata. For details on how to setup your upload routines, refer to the [Megalista Wiki](https://github.com/google/megalista/wiki) or the [Megalista user guide](https://github.com/google/megalista/blob/main/documentation/Megalista%20-%20Technical%20User%20Guide%20-%20EXTERNAL.pdf).
+Every upload method expects as source a BigQuery data with specific fields, in addition to specific configuration metadata. For details on how to setup your upload routines, refer to the [MDS Wiki](https://github.com/dp6/marketing-data-sync/wiki) or the [MDS user guide](https://github.com/dp6/marketing-data-sync/blob/main/documentation/mds%20-%20Technical%20User%20Guide%20-%20EXTERNAL.pdf).
+
+### Mandatory requirements
+
+Only contributions that meet the following requirements will be accepted:
+
+- [Commit pattern](https://www.conventionalcommits.org/en/v1.0.0/)
+
+## Support:
+
+**DP6 Koopa-troopa Team**
+
+_e-mail: <[email protected]>_
+
+<img src="https://raw.githubusercontent.com/DP6/templates-centro-de-inovacoes/main/public/images/koopa.png" height="100" />

_config.yml

@@ -0,0 +1,8 @@
+title: DP6 - Centro de inovações
+initiative: 'Marketing Data Sync'
+
+remote_theme: dp6/dp6.github.io
+
+plugins:
+  - jekyll-sitemap
+  - jekyll-gzip

cloud_config/scheduler_sample.json

@@ -1,17 +1,17 @@
 {
-    "jobName": "megalist_daily",
+    "jobName": "mds_daily",
     "parameters": {
         "developer_token": "Google Ads Developer Token",
         "client_id": "GCP OAuth Client id",
         "client_secret": "GCP OAuth Client Secret",
         "access_token": "GCP OAuth access token",
         "refresh_token": "GCP OAuth refresh token",
         "setup_sheet_id": "Setup Google Sheets Id",
-        "bq_ops_dataset": "Auxliary bigquery dataset used for Megalista operations",
+        "bq_ops_dataset": "Auxliary bigquery dataset used for MDS operations",
 	"appsflyer_dev_key": "Apps flyer dev key"
     },
     "environment": {
-        "tempLocation": "gs://megalist-data/temp",
+        "tempLocation": "gs://bucket-name/temp",
         "zone": "us-central1-f"
     }
 }

megalist_dataflow/uploaders/utils.py

@@ -32,7 +32,7 @@ def get_ads_service(service_name, version, oauth_credentials, developer_token,
     client = adwords.AdWordsClient(
         developer_token,
         oauth2_client,
-        'MegaList Dataflow',
+        'Mds Dataflow',
         client_customer_id=customer_id)
     return client.GetService(service_name, version=version)
 

run_cloud.sh

@@ -21,4 +21,4 @@ fi
 
 gcloud config set project $1
 token=$(gcloud auth application-default print-access-token)
-curl -H "Authorization: Bearer $token" -H "Content-Type:application/json" "https://dataflow.googleapis.com/v1b3/projects/$1/locations/$3/templates:launch?gcsPath=gs://$2/templates/megalist" --data-binary "@cloud_config/scheduler.json"
+curl -H "Authorization: Bearer $token" -H "Content-Type:application/json" "https://dataflow.googleapis.com/v1b3/projects/$1/locations/$3/templates:launch?gcsPath=gs://$2/templates/mds" --data-binary "@cloud_config/scheduler.json"