Skip to content

Commit

Permalink
Merge pull request #86 from fvst-dev/feature/more-documentation
Browse files Browse the repository at this point in the history
chore: updated README.md
  • Loading branch information
peeter-tomberg authored Aug 20, 2023
2 parents 0e537e0 + 9a9e961 commit 6704cd9
Show file tree
Hide file tree
Showing 7 changed files with 227 additions and 73 deletions.
65 changes: 7 additions & 58 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,65 +7,12 @@ This repositories goal is to simplify microservices architecture and enable comp

We use GCP as the cloud provider - it provides an extensive free tier to get started - https://cloud.google.com/free/

## Getting started
## Documentation

### Copy this repository template

**1. Click "Use this template"**

![](https://ajeuwbhvhr.cloudimg.io/colony-recorder.s3.amazonaws.com/files/2023-08-17/2a6445cf-d3a9-4765-bd37-fe333c8df258/ascreenshot.jpeg?tl_px=1736,324&br_px=3456,1285&force_format=png&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=538,276)

**2. Click Create new repository**

![](https://ajeuwbhvhr.cloudimg.io/colony-recorder.s3.amazonaws.com/files/2023-08-17/c3ba8cd7-c5f3-44cf-9618-aa7a9f548ba4/ascreenshot.jpeg?tl_px=1736,372&br_px=3456,1333&force_format=png&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=540,277)

**3. Include all branches**

![](https://ajeuwbhvhr.cloudimg.io/colony-recorder.s3.amazonaws.com/files/2023-08-17/0c0d572c-75a1-4c5f-aa48-dda5841d7b77/ascreenshot.jpeg?tl_px=1245,561&br_px=2965,1522&force_format=png&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=524,277)

**4. Give your repository a name**

![](https://ajeuwbhvhr.cloudimg.io/colony-recorder.s3.amazonaws.com/files/2023-08-17/3539b28c-8db7-444b-b199-c789ab218192/ascreenshot.jpeg?tl_px=1457,759&br_px=3177,1720&force_format=png&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=524,277)

**6. Click "Create repository"**

**7. Checkout your repository**

### Install global dependencies

- Github CLI - `brew install gh` - Used for managing secrets and variables in Github actions
- Google cloud SDK - `brew install --cask google-cloud-sdk` - Used to setup projects on GCP.
- Make sure the SDK is up to date by running `gcloud components update`
- Terraform `brew install terraform` - Used to manage infrastructure as code

### Setup GCP

- `npm install`
- `npx fvst infra init` - This CLI script will guide you through setting up the infrastructure on GCP.
- You have to set up a billing account - follow the tutorial at https://cloud.google.com/billing/docs/how-to/create-billing-account - this step asks for a credit card, but GCP will not charge the account when the free trial runs out.
- This step will take some time, it will create 2 projects (staging/production), build docker containers and deploy them.
- Setup a clerk.com account following the tutorial at [Setup Clerk application](./docs/clerk/setup_clerk_application.md)
- You should have one account for staging and one account for production
- Wait for the `npx fvst infra init` script and the workflow it starts on GitHub actions to finish
- The workflow will fail - we're unsure why it fails, it seems like GCP IAM policy is not applied on the first run.
- This is ok, we still need to configure the secrets for Clerk
- `npx fvst infra setup-secrets staging` - This scripts updates the secret values for the staging environment, you should do the same for production.
- [How to get the CLERK_ISSUER value](./docs/clerk/get_clerk_issuers_value.md)
- [How to get the CLERK_JWSK_URL value](./docs/clerk/get_clerk_jwsk_url_value.md)
- [How to get the CLERK_PUBLISHABLE_KEY value](./docs/clerk/get_clerk_publishable_key_value.md)
- [How to get the CLERK_SECRET_KEY value](./docs/clerk/get_clerk_secret_key_value.md)
- Rerun the failed jobs from the Github Actions workflow
- Open `https://console.cloud.google.com/welcome/`
- Choose one of the projects created from the top
- Choose `Cloud run` from the navigation menu on the left
- You should be greeted with the following screen ![](https://gcdnb.pbrd.co/images/hlHko6NmevNp.png?o=1)

### Setup local development

- `npx fvst dev init` - This scripts loads the secret values from staging and populates the .env files with them for local development
- `docker compose up` - Brings up postgres & redis
- `npm run dev` - brings up all applications
- Go to `http://localhost:3000/` and you should be greeted with the following screen ![](https://gcdnb.pbrd.co/images/qfIur8N7nytk.png?o=1)
- [Setup](./docs/setup/README.md)
- [Infrastructure overview](./docs/infra/README.md)
- [Branching](./docs/branching/README.md)
- [CI](./docs/ci/README.md)

## Folder structure

Expand All @@ -78,6 +25,8 @@ logic.
| apps | All backend and frontend apps |
| packages | Here we keep code that we want to re-use across apps |
| tooling | Here we have all the configuration code for all the tooling we use inside the monorepository |
| infra | Here we have all the infrastructure configuration code to deploy to GCP |
| docs | Documentation |

## NPM Scripts

Expand Down
68 changes: 68 additions & 0 deletions docs/branching/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Branching / release strategy

You can use both release train or feature based releases

## Release train

For release train, you start a new feature branch from `develop`. Once you're ready to release your code you make a pull request against `develop`.
When your code is merged to `develop`, it is automatically deployed to the `staging` environment. You validate your code on there and once everyone is happy you merge
develop into `main` to release to `production`. A `production` release can include multiple features. It's done periodically, e.g. once an hour, once a day, once a week.

This strategy is simple, but has the downside that one feature might prevent another from being released (e.g. `develop` has two features merged into it, feature A is ready to go but feature B has bugs)

```mermaid
gitGraph
commit id: "Initial ocommit"
branch develop
branch feature/JIRA-1
checkout feature/JIRA-1
commit id: "feat(one): Initial commit"
commit id: "chore(one): Bug fixes"
commit id: "docs(one): Added docs"
checkout develop
merge feature/JIRA-1 tag:"Release to staging"
branch feature/JIRA-2
checkout feature/JIRA-2
commit id: "feat(two): Initial commit"
commit id: "chore(two): Bug fixes"
commit id: "docs(two): Added docs"
checkout develop
merge feature/JIRA-2 tag:"Release to staging"
checkout main
merge develop tag:"Release to production"
```

## Feature releases

For feature releases, you start your branch from `main`. Once your code is ready, you open a pull request against `develop` and once its merged it is automatically deployed to `staging`.
Once you've validated your code on `staging`, you create a new pull request from your feature branch to `main`. Once it's merged, your changes are automatically deployed to `production`.

This strategy is more complex as it requires multiple pull requests and needs a separate flow to sync `main` and `develop` branches after a feature release, but it allows each release to be deployed independently.

```mermaid
gitGraph
commit id: "Initial commit"
branch develop
checkout main
commit type:normal id:"Start JIRA-1"
checkout main
branch feature/JIRA-1
checkout feature/JIRA-1
commit id: "feat(one): Initial commit"
commit id: "chore(one): Bug fixes"
commit id: "docs(one): Added docs"
checkout develop
merge feature/JIRA-1 tag:"Release to staging"
checkout main
merge feature/JIRA-1 tag:"Release to production"
commit type:normal id:"Start JIRA-2"
branch feature/JIRA-2
checkout feature/JIRA-2
commit id: "feat(two): Initial commit"
commit id: "chore(two): Bug fixes"
commit id: "docs(two): Added docs"
checkout develop
merge feature/JIRA-2 tag:"Release to staging"
checkout main
merge feature/JIRA-2 tag:"Release to production"
```
10 changes: 10 additions & 0 deletions docs/ci/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# CI/CD

## Workflows

| Workflow | Trigger | Description |
| ------------------------------------------------------------ | ------------------------- | ------------------------------------------------------------------------------------------------------------ |
| [initialize](../../.github/workflows/initialize.yml) | Manual trigger | Used by the CLI to build containers and deploy the infrastructure |
| [deploy](../../.github/workflows/deploy.yml) | Push on main / develop | Builds new docker containers for all services and pushes to appropriate environment |
| [pull-request](../../.github/workflows/pull-request.yml) | PR against main / develop | Runs linters and test suite on each PR against main and develop |
| [terraform-plan](../../.github/workflows/terraform-plan.yml) | PR against main / develop | Runs terraform plan on each PR against main and develop and leaves the plan as a message on the pull request |
File renamed without changes.
83 changes: 83 additions & 0 deletions docs/infra/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Infrastructure

We use terraform to set up 2 projects on GCP - staging and production.

## Folder structure

| Directory | What goes inside of it |
| ----------------- | ----------------------------------------------------------------------------------------------------- |
| ./infra/envs/ | All environment definitions go here. What services are configured and how they depends on each other. |
| ./infra/modules/ | All reusable modules go here, e.g. google cloud run definitions. |
| ./infra/services/ | Here we have all of the service definitions |

## Infrastructure overview

```mermaid
C4Context
title Infrastructure overview
Person(user, "User")
Boundary(internet, "Internet") {
System(Web, "Web", "NextJS application")
System(Gateway, "Graphql Gateway", "NestJS application")
}
Boundary(services, "Services") {
System(Worker, "demo-worker", "NestJS application")
System(User, "user-graphql", "NestJS application")
System(Blog, "blog-graphql", "NestJS application")
System(Todo, "todo-graphql", "NestJS application")
}
Boundary(db, "Storage") {
SystemDb(Redis, "Managed Redis", "Memorystore instance")
SystemDb(UserSql, "User database", "Cloud sql instance")
SystemDb(BlogSql, "Blog database", "Cloud sql instance")
SystemDb(TodoSql, "Todo database", "Cloud sql instance")
}
Rel(user, Web, "Visits", "HTTP")
Rel(Web, Gateway, "Requestes data", "HTTP/Graphql")
Rel(Gateway, User, "Requestes data", "HTTP/Graphql")
Rel(Gateway, Blog, "Requestes data", "HTTP/Graphql")
Rel(Gateway, Todo, "Requestes data", "HTTP/Graphql")
BiRel(User, UserSql, "Read/write", "Prisma")
BiRel(Blog, BlogSql, "Read/write", "Prisma")
BiRel(Todo, TodoSql, "Read/write", "Prisma")
Rel(User, Redis, "Read/write", "Bull MQ")
Rel(Blog, Redis, "Read/write", "Bull MQ")
Rel(Todo, Redis, "Read/write", "Bull MQ")
BiRel(Worker, Redis, "Read/write", "Bull MQ")
UpdateRelStyle(user, Web, $textColor="blue", $lineColor="black", $offsetX="-20", $offsetY="-40")
UpdateRelStyle(Web, Gateway, $textColor="blue", $lineColor="black", $offsetX="-40", $offsetY="-40")
UpdateRelStyle(Gateway, User, $textColor="blue", $lineColor="gold", $offsetX="-40", $offsetY="-40")
UpdateRelStyle(Gateway, Blog, $textColor="blue", $lineColor="gold", $offsetX="-80", $offsetY="-40")
UpdateRelStyle(Gateway, Todo, $textColor="blue", $lineColor="gold", $offsetX="-120", $offsetY="-40")
UpdateRelStyle(User, UserSql, $textColor="blue", $lineColor="blue", $offsetX="-30", $offsetY="-60")
UpdateRelStyle(Blog, BlogSql, $textColor="blue", $lineColor="blue", $offsetX="-30", $offsetY="-60")
UpdateRelStyle(Todo, TodoSql, $textColor="blue", $lineColor="blue", $offsetX="-30", $offsetY="-60")
UpdateElementStyle(UserSql, $bgColor="green")
UpdateElementStyle(BlogSql, $bgColor="green")
UpdateElementStyle(TodoSql, $bgColor="green")
UpdateRelStyle(User, UserSql, $lineColor="green")
UpdateRelStyle(Blog, BlogSql, $lineColor="green")
UpdateRelStyle(Todo, TodoSql, $lineColor="green")
UpdateElementStyle(Redis, $bgColor="orange")
UpdateRelStyle(Worker, Redis, $lineColor="orange", $textColor="blue", $offsetX="-50")
UpdateRelStyle(User, Redis, $lineColor="orange", $textColor="blue", $offsetX="-30")
UpdateRelStyle(Blog, Redis, $lineColor="orange", $textColor="blue", $offsetX="0")
UpdateRelStyle(Todo, Redis, $lineColor="orange", $textColor="blue", $offsetX="40")
UpdateLayoutConfig($c4ShapeInRow="10", $c4BoundaryInRow="1")
```

[Open editor](https://mermaid.live/edit#pako:eNq1V1tv2jAU_iuR1QcmQRtIChRpL6VT12mdOtLuJl5McqDREps6Timr-O87jhMI4RbSLRKKMef7vuNzMfYrcbkHpEf6dp8zCS9yyAx8pC8DMG7YWNBIitiVsQCDP4N49mGmTe5ARJzV4ghE3RiSB3wPyTvDGDL9-yWPmUfFvOYjr2AgldVNOlaWr9pOPc48khDWvsNIGeFrSNTgC_rzyTHodBr4LpU-Z6lCAXdNJczoXEGuBZ0-PgVGOpXxRLt5FhsOR2qZLkQK6qTjHQ5z8Vsv34OQN2bJ172a6tmgeUhjqGLZmOgVlPA8x3AZ8ImyH-G7GsM997iyl_g-gmEzel6SQ0dyQSewNWxXo9oAPD-J7y1laOYZyYTWu8VQCjTkWHI-iyRlLmz1GXlU5JynICtAw6OSjmgEmqkf8NgzIiyHQzwqfimPGlbmUVFMedTwGJ4kkno4gCDtq7QjvvmRL9PwfLy_v0PYyjKxybXAAJ5iTBlEifoKdHadZTUHXuKyGqyGzuqvGjqrvVLotNx8xaCdzhXBAKh3NhO-TCN-J_wopEtVjdLO5lJeCqWdzCV4D2q1Ru3hstyLmMs4CIzbr2th0e4dBdG-lYboBWWbVymYBj5MMTGAYEfOA8jX6In67-jzgIv3aheKE4qTwGewmqTu72SWj8cRyB8412iZuZmfasY2l24W1NbrvKKgvUtwq2ShOw5rTnjglZbcJ6iLoJpgt4qgLqFqgs3WUVEtdG2ZTKaTOU1rQ7K9c5GFhv__goW94h8Lrof1QwAhMLkKrZYcTZYZEwCs4OkaaBWYI0Crxe0H7cz6WlHtAxazVxpYzMJ24M4Vpvtibn14pGHJmWb7_rS-n67JZchdtZBL-7m5P4JvpLcOdMnb6A90xNvIc7tYXuMznfMYcWzsT2onru080incsAGfIaapm8i1s_Pp8oeEi9RJCCKkvod3oOSgiifgRyyCIeklp_oxjQO8rQzZAk1pLLkzZy7p4Z0I6iRO9K98iuflkPTGNIhwdkrZL87DzAi_kt4reSG9Rss-7Z63mhfmudW18WO162ROevbFqdWx2u1mt2mZnU7HXtTJn4SheWqanXbXbllm17Rt27LrBEOIR-NbfW1Lbm-Lvz9ZZEs)
59 changes: 59 additions & 0 deletions docs/setup/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Getting started

## Copy this repository template

**1. Click "Use this template"**

![](https://ajeuwbhvhr.cloudimg.io/colony-recorder.s3.amazonaws.com/files/2023-08-17/2a6445cf-d3a9-4765-bd37-fe333c8df258/ascreenshot.jpeg?tl_px=1736,324&br_px=3456,1285&force_format=png&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=538,276)

**2. Click Create new repository**

![](https://ajeuwbhvhr.cloudimg.io/colony-recorder.s3.amazonaws.com/files/2023-08-17/c3ba8cd7-c5f3-44cf-9618-aa7a9f548ba4/ascreenshot.jpeg?tl_px=1736,372&br_px=3456,1333&force_format=png&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=540,277)

**3. Include all branches**

![](https://ajeuwbhvhr.cloudimg.io/colony-recorder.s3.amazonaws.com/files/2023-08-17/0c0d572c-75a1-4c5f-aa48-dda5841d7b77/ascreenshot.jpeg?tl_px=1245,561&br_px=2965,1522&force_format=png&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=524,277)

**4. Give your repository a name**

![](https://ajeuwbhvhr.cloudimg.io/colony-recorder.s3.amazonaws.com/files/2023-08-17/3539b28c-8db7-444b-b199-c789ab218192/ascreenshot.jpeg?tl_px=1457,759&br_px=3177,1720&force_format=png&width=1120.0&wat=1&wat_opacity=0.7&wat_gravity=northwest&wat_url=https://colony-recorder.s3.us-west-1.amazonaws.com/images/watermarks/FB923C_standard.png&wat_pad=524,277)

**6. Click "Create repository"**

**7. Checkout your repository**

## Install global dependencies

- Github CLI - `brew install gh` - Used for managing secrets and variables in Github actions
- Google cloud SDK - `brew install --cask google-cloud-sdk` - Used to setup projects on GCP.
- Make sure the SDK is up to date by running `gcloud components update`
- Terraform `brew install terraform` - Used to manage infrastructure as code

## Setup GCP

- `npm install`
- `npx fvst infra init` - This CLI script will guide you through setting up the infrastructure on GCP.
- You have to set up a billing account - follow the tutorial at https://cloud.google.com/billing/docs/how-to/create-billing-account - this step asks for a credit card, but GCP will not charge the account when the free trial runs out.
- This step will take some time, it will create 2 projects (staging/production), build docker containers and deploy them.
- Setup a clerk.com account following the tutorial at [Setup Clerk application](../clerk/README.md)
- You should have one account for staging and one account for production
- Wait for the `npx fvst infra init` script and the workflow it starts on GitHub actions to finish
- The workflow will fail - we're unsure why it fails, it seems like GCP IAM policy is not applied on the first run.
- This is ok, we still need to configure the secrets for Clerk
- `npx fvst infra setup-secrets staging` - This scripts updates the secret values for the staging environment, you should do the same for production.
- [How to get the CLERK_ISSUER value](../clerk/get_clerk_issuers_value.md)
- [How to get the CLERK_JWSK_URL value](../clerk/get_clerk_jwsk_url_value.md)
- [How to get the CLERK_PUBLISHABLE_KEY value](../clerk/get_clerk_publishable_key_value.md)
- [How to get the CLERK_SECRET_KEY value](../clerk/get_clerk_secret_key_value.md)
- Rerun the failed jobs from the Github Actions workflow
- Open `https://console.cloud.google.com/welcome/`
- Choose one of the projects created from the top
- Choose `Cloud run` from the navigation menu on the left
- You should be greeted with the following screen ![](https://gcdnb.pbrd.co/images/hlHko6NmevNp.png?o=1)

## Setup local development

- `npx fvst dev init` - This scripts loads the secret values from staging and populates the .env files with them for local development
- `docker compose up` - Brings up postgres & redis
- `npm run dev` - brings up all applications
- Go to `http://localhost:3000/` and you should be greeted with the following screen ![](https://gcdnb.pbrd.co/images/qfIur8N7nytk.png?o=1)
15 changes: 0 additions & 15 deletions infra/README.md

This file was deleted.

0 comments on commit 6704cd9

Please sign in to comment.