Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove 'warm' build steps from the template pipelines #1223

Closed
wants to merge 6 commits into from
Closed

Remove 'warm' build steps from the template pipelines #1223

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
83d69ee
remove 'warm' build steps
gitlon Jul 19, 2023
2b4834c
remove 'warm' build steps
gitlon Jul 19, 2023
76f6555
remove 'warm' build steps
gitlon Jul 19, 2023
f4d32c5
remove 'warm' build steps
gitlon Jul 19, 2023
5001cfc
Merge branch 'master' into master
gitlon Jul 19, 2023
7991fed
Merge branch 'seek-oss:master' into master
gitlon Aug 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 0 additions & 10 deletions template/express-rest-api/.buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,21 +30,12 @@ configs:
permit_on_passed: true

steps:
- label: 🧖‍♀️ Warm Prod
command: ':'
key: warm-prod
plugins:
- *aws-sm
- *private-npm
- *docker-ecr-cache
Comment on lines -33 to -39
Copy link
Member

@72636c 72636c Jul 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intent here was to bake the Docker build image and upload it to ECR once, rather than doing it twice on Test & Lint and Build & Package. Thoughts?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah in my team's repo we call the step build cache

Copy link
Contributor Author

@gitlon gitlon Jul 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes of course that is still valid.

How about serialising the two steps, so it gets done only once - putting the shortest step first (which I think will be Build & Package)? Or am I just wrong, and the warm step should remain.

Copy link
Contributor

@samchungy samchungy Jul 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If both run at the same time there would be a chance we would get a cache miss right? The build cache step takes a little while

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By "serialise" I meant adding a dependency so that one step always runs after the other, so no chance of a cache miss

Copy link
Contributor

@samchungy samchungy Jul 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not to mention it might fail while quicker while running both lint and build at the same time = faster feedback

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am investigating / gathering stats.

Copy link
Contributor

@samchungy samchungy Jul 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, as of the new build vCurrent strategy it looks like each step has to do the pull image step, so the question is whether that pull image step is quicker than running the commands on a single step?

Copy link
Contributor

@AaronMoat AaronMoat Jul 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if we could somehow lock the ecr cache plugin to only run once per build at a time, then the first one that runs will build + the others will pull once it's finished pushing the image, no need for a warm step. Though I suppose this wouldn't actually save any time (the agent would just have to idle), right?

Copy link
Contributor

@samchungy samchungy Sep 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This definitely helps in UI repos, given the UI images are typically alot bigger.

It takes on average 1 min 30 seconds to pull the image on one of our UI repos so moving all the commands to the single step has sped it up a bit


- label: 🧪 Test & Lint
commands:
- echo '+++ yarn test:ci'
- yarn test:ci
- echo '--- yarn lint'
- yarn lint
depends_on: warm-prod
env:
GET_GITHUB_TOKEN: please
plugins:
Expand All @@ -56,7 +47,6 @@ steps:
timeout_in_minutes: 10

- label: 📦 Build & Package
depends_on: warm-prod
plugins:
- *aws-sm
- *private-npm
Expand Down
10 changes: 0 additions & 10 deletions template/koa-rest-api/.buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,21 +30,12 @@ configs:
permit_on_passed: true

steps:
- label: 🧖‍♀️ Warm Prod
command: ':'
key: warm-prod
plugins:
- *aws-sm
- *private-npm
- *docker-ecr-cache

- label: 🧪 Test & Lint
commands:
- echo '+++ yarn test:ci'
- yarn test:ci
- echo '--- yarn lint'
- yarn lint
depends_on: warm-prod
env:
GET_GITHUB_TOKEN: please
plugins:
Expand All @@ -56,7 +47,6 @@ steps:
timeout_in_minutes: 10

- label: 📦 Build & Package
depends_on: warm-prod
plugins:
- *aws-sm
- *private-npm
Expand Down
10 changes: 0 additions & 10 deletions template/lambda-sqs-worker-cdk/.buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,16 +61,6 @@ steps:
run: app
timeout_in_minutes: 10

- agents:
queue: <%- devBuildkiteQueueName %>
branches: '!renovate-*'
label: 🧖‍♀️ Warm Dev
command: ':'
plugins:
- *aws-sm
- *private-npm
- *docker-ecr-cache

- wait
- block: 🙋🏻‍♀️ Deploy Dev
branches: '!${BUILDKITE_PIPELINE_DEFAULT_BRANCH}'
Expand Down
10 changes: 0 additions & 10 deletions template/lambda-sqs-worker/.buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,16 +64,6 @@ steps:
run: app
timeout_in_minutes: 10

- agents:
queue: <%- devBuildkiteQueueName %>
branches: '!renovate-*'
label: 🧖‍♀️ Warm Dev
command: ':'
plugins:
- *aws-sm
- *private-npm
- *docker-ecr-cache

- wait
- block: 🙋🏻‍♀️ Deploy Dev
branches: '!${BUILDKITE_PIPELINE_DEFAULT_BRANCH}'
Expand Down