SWC-6624: store Playwright reports with traces in S3 when e2e workflow runs in Sage repo #5245
Conversation
…g in Sage repo, otherwise continue attaching report without trace as GitHub artifact
…eports stored in s3
| # Ensure that at most one playwright-tests job will run at a time in Sage-Bionetworks repo, | ||
| # but allow multiple playwright-test jobs to run concurrently in forked repos | ||
| # Per https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#concurrency | ||
| # "When a concurrent job or workflow is queued, if another job or workflow | ||
| # using the same concurrency group in the repository is in progress, | ||
| # the queued job or workflow will be pending. Any previously pending job or | ||
| # workflow in the concurrency group will be cancelled." | ||
| # Related discussion here: https://github.com/orgs/community/discussions/5435 | ||
| concurrency: | ||
| group: ${{ github.repository_owner == 'Sage-Bionetworks' && format('${0}-${1}', github.workflow, '-playwright-tests') || format('${0}-${1}', github.run_id, matrix.shard ) }} |
There was a problem hiding this comment.
Apply a concurrency group to playwright-tests jobs so that the e2e workflow uses at most one macOS runner at a time. GitHub's current concurrency implementation cancels previously queued pending jobs, rather than queuing and executing all pending jobs. So if more than two e2e workflow runs are triggered at the same time, depending on when each playwright-tests job starts, a subset or none of the e2e shards may run per workflow run. However, once all other jobs have finished running, the cancelled job(s) could be manually re-triggered, if needed.
For now, plan to monitor the e2e tests and the frequency of cancelled jobs, then decide whether another solution should be investigated.
Note: I applied concurrency at the playwright-tests job level, so that the build job (which builds SWC and runs unit tests on a Linux runner) will always run, regardless of how many e2e workflows have been triggered.
| TRACE_TOGGLE: ${{ github.repository_owner == 'Sage-Bionetworks' && 'on' || 'off'}} | ||
| run: yarn playwright test --shard ${{ matrix.shard }} --trace=${{ env.TRACE_TOGGLE }} |
There was a problem hiding this comment.
Only record a trace when running the workflow in the Sage-Bionetworks repo
| - name: Upload blob report to GitHub Actions Artifacts | ||
| uses: actions/upload-artifact@v3 | ||
| if: always() | ||
| if: ${{ github.repository_owner != 'Sage-Bionetworks' && always() }} |
There was a problem hiding this comment.
Only attach the blob report as a GitHub artifact in forked repos
| @@ -83,7 +112,7 @@ jobs: | |||
| merge-reports: | |||
| # Merge reports after playwright-tests, even if some shards have failed | |||
| # But skip this job if the previous job was cancelled or skipped | |||
| if: ${{ !cancelled() && needs.playwright-tests.result != 'skipped' }} | |||
| if: ${{ github.repository_owner != 'Sage-Bionetworks' && !cancelled() && needs.playwright-tests.result != 'skipped' }} | |||
There was a problem hiding this comment.
Similarly, only merge the blob reports into an HTML report in forked repos
| - name: Upload blob report to S3 | ||
| if: github.repository_owner == 'Sage-Bionetworks' | ||
| run: | | ||
| aws s3 sync ./blob-report --region us-east-1 s3://${{ env.DESTINATION_BUCKET }}/${{ env.REPORT_ID }}/ |
There was a problem hiding this comment.
Would a retention policy be specified here? Or is it specified on the bucket?
There was a problem hiding this comment.
The policy is specified on the bucket -- objects are set to expire after 14 days, which matches the retention set for reports stored as GitHub artifacts.
| -H "Accept: application/vnd.github+json" \ | ||
| -H "X-GitHub-Api-Version: 2022-11-28" \ | ||
| "/repos/${REPO_OWNER}/${REPO_NAME}/actions/runs/${RUN_ID}/artifacts" \ | ||
| | jq '.artifacts[] | select(.name | contains("html-report")) | {id, name}') |
There was a problem hiding this comment.
jq may not be installed on dev machines, might want to add that to instructions (unless maybe it's a dependency of gh?)
There was a problem hiding this comment.
Oh good call! I don't believe jq is a dependency of gh, so I'll make a note to install it in the README.
Co-authored-by: Nick Grosenbacher <[email protected]>
developandrelease-**branchesplaywright-testsjob to run at a time within the repo