Store screenshots of CI e2e failures as CI artifacts #26664

adamziel · 2020-11-03T16:38:53Z

Description

This PR updates e2e testing workflow to ensure test failures are captured on screenshots for easier debugging.

Testing instructions

This PR purposefully breaks a test to trigger screenshot capture. To see if it works, go to e2e test failures and confirm you can see the following artifact:

github-actions · 2020-11-03T16:47:52Z

Size Change: +174 B (0%)

Total Size: 1.19 MB

Filename	Size	Change
`build/block-editor/index.js`	133 kB	+15 B (0%)
`build/block-library/index.js`	147 kB	-9 B (0%)
`build/components/style.css`	15.3 kB	+1 B
`build/edit-site/index.js`	23 kB	+164 B (0%)
`build/edit-site/style-rtl.css`	3.95 kB	+2 B (0%)
`build/edit-site/style.css`	3.95 kB	+1 B

ℹ️ View Unchanged

Filename	Size	Change
`build/a11y/index.js`	1.14 kB	0 B
`build/annotations/index.js`	3.77 kB	0 B
`build/api-fetch/index.js`	3.42 kB	0 B
`build/autop/index.js`	2.83 kB	0 B
`build/blob/index.js`	664 B	0 B
`build/block-directory/index.js`	8.71 kB	0 B
`build/block-directory/style-rtl.css`	943 B	0 B
`build/block-directory/style.css`	942 B	0 B
`build/block-editor/style-rtl.css`	11.1 kB	0 B
`build/block-editor/style.css`	11.1 kB	0 B
`build/block-library/editor-rtl.css`	8.91 kB	0 B
`build/block-library/editor.css`	8.91 kB	0 B
`build/block-library/style-rtl.css`	8.1 kB	0 B
`build/block-library/style.css`	8.1 kB	0 B
`build/block-library/theme-rtl.css`	792 B	0 B
`build/block-library/theme.css`	793 B	0 B
`build/block-serialization-default-parser/index.js`	1.87 kB	0 B
`build/block-serialization-spec-parser/index.js`	3.06 kB	0 B
`build/blocks/index.js`	48 kB	0 B
`build/components/index.js`	171 kB	0 B
`build/components/style-rtl.css`	15.3 kB	0 B
`build/compose/index.js`	9.89 kB	0 B
`build/core-data/index.js`	14.8 kB	0 B
`build/data-controls/index.js`	821 B	0 B
`build/data/index.js`	8.74 kB	0 B
`build/date/index.js`	11.2 kB	0 B
`build/deprecated/index.js`	768 B	0 B
`build/dom-ready/index.js`	571 B	0 B
`build/dom/index.js`	4.52 kB	0 B
`build/edit-navigation/index.js`	11.1 kB	0 B
`build/edit-navigation/style-rtl.css`	881 B	0 B
`build/edit-navigation/style.css`	885 B	0 B
`build/edit-post/index.js`	306 kB	0 B
`build/edit-post/style-rtl.css`	6.43 kB	0 B
`build/edit-post/style.css`	6.42 kB	0 B
`build/edit-widgets/index.js`	26.3 kB	0 B
`build/edit-widgets/style-rtl.css`	3.16 kB	0 B
`build/edit-widgets/style.css`	3.16 kB	0 B
`build/editor/editor-styles-rtl.css`	476 B	0 B
`build/editor/editor-styles.css`	478 B	0 B
`build/editor/index.js`	42.5 kB	0 B
`build/editor/style-rtl.css`	3.85 kB	0 B
`build/editor/style.css`	3.85 kB	0 B
`build/element/index.js`	4.62 kB	0 B
`build/escape-html/index.js`	735 B	0 B
`build/format-library/index.js`	6.86 kB	0 B
`build/format-library/style-rtl.css`	547 B	0 B
`build/format-library/style.css`	548 B	0 B
`build/hooks/index.js`	2.16 kB	0 B
`build/html-entities/index.js`	623 B	0 B
`build/i18n/index.js`	3.57 kB	0 B
`build/is-shallow-equal/index.js`	713 B	0 B
`build/keyboard-shortcuts/index.js`	2.52 kB	0 B
`build/keycodes/index.js`	1.94 kB	0 B
`build/list-reusable-blocks/index.js`	3.1 kB	0 B
`build/list-reusable-blocks/style-rtl.css`	476 B	0 B
`build/list-reusable-blocks/style.css`	476 B	0 B
`build/media-utils/index.js`	5.31 kB	0 B
`build/notices/index.js`	1.77 kB	0 B
`build/nux/index.js`	3.4 kB	0 B
`build/nux/style-rtl.css`	671 B	0 B
`build/nux/style.css`	668 B	0 B
`build/plugins/index.js`	2.56 kB	0 B
`build/primitives/index.js`	1.43 kB	0 B
`build/priority-queue/index.js`	790 B	0 B
`build/redux-routine/index.js`	2.83 kB	0 B
`build/reusable-blocks/index.js`	3.05 kB	0 B
`build/rich-text/index.js`	13.3 kB	0 B
`build/server-side-render/index.js`	2.77 kB	0 B
`build/shortcode/index.js`	1.69 kB	0 B
`build/token-list/index.js`	1.27 kB	0 B
`build/url/index.js`	4.06 kB	0 B
`build/viewport/index.js`	1.83 kB	0 B
`build/warning/index.js`	1.14 kB	0 B
`build/wordcount/index.js`	1.22 kB	0 B

_{compressed-size-action}

talldan · 2020-11-05T08:37:03Z

This would be great. Any thoughts on the best way to handle storing the images? Committing image files in git usually isn't a great option.

I think I remember seeing that github can work with git-lfs, which might be an option, not sure if there's a cost involved:
https://github.blog/2015-04-08-announcing-git-large-file-storage-lfs/

edit: commented too soon, I see this is using artifacts.

adamziel · 2020-11-05T12:43:15Z

@talldan yup I'm trying artifacts for the first pass, I'm open to other options as well. For now I'm just trying it to take the screenshots, somehow the obvious solutions don't seem to work.

adamziel · 2020-11-10T21:24:47Z

@talldan it worked!

cc @noisysocks @karmatosed @jasmussen @draganescu @tellthemachines @kevin940726 @jorgefilipecosta

packages/e2e-tests/config/setup-screenshots.js

noahtallen

This is amazing!!! I was curious, have we looked into storing screen recordings as well?

At any rate, I'll approve it from my point of view since I don't see anything out of place and this will be very helpful. Just left some nitpicks about the promises

packages/e2e-tests/config/setup-screenshots.js

noahtallen · 2020-11-10T22:58:56Z

packages/e2e-tests/config/setup-screenshots.js

+}
+
+/**
+ * @copyright Tom Esterez (@testerez) https://github.com/smooth-code/jest-puppeteer/issues/131#issuecomment-424073620


oh, also any concerns about copyright or licensing here?

After your review I ended up starting from scratch so probably not, however there is still a degree of inspiration so I believe an acknowledgement note would be a nice thing to do.

CC @testerez - your idea from argos-ci/jest-puppeteer#131 (comment) is pretty awesome and inspired this PR, we would like to include an acknowledgement note - thoughts? :-)

Appreciate it, thanks!

noisysocks · 2020-11-11T00:05:08Z

Thanks for doing this! ❤️

What do you think about dumping document.body.innerHTML into a .html artefact as well? I think it will help with debugging "selector doesn't exist" failures to see what the exact state of the DOM is at the time of failure.

jasmussen · 2020-11-11T08:07:25Z

Really cool, going to be amazingly helpful. Props!

adamziel · 2020-11-12T13:40:06Z

@noisysocks I added capturing HTML snapshots :-)
@noahtallen recording is an interesting idea! Let's explore that in a separate PR, I'm concerned about the storage. This package that wraps ffmpeg to record the browser could come handy: https://github.com/clipisode/puppeteer-recorder/blob/master/index.js

packages/e2e-tests/config/setup-debug-artifacts.js

gziolo · 2020-11-12T18:20:24Z

This is awesome, great addition @adamziel. Major props 💯

ellatrix · 2020-12-07T09:01:12Z

Hi @adamziel! Do you have any idea how to add this to the performance tests? I tried adding it, but no artefacts are created. I have a failure in #25775.

adamziel · 2020-12-07T09:19:37Z

@ellatrix I think what's missing is the setup bit, note how e2e tests use jest.config.js:

$( npm bin )/wp-scripts test-e2e --config=./packages/e2e-tests/jest.config.js

while performance tests don't:

./bin/plugin/cli.js perf --ci $GITHUB_SHA master --tests-branch $GITHUB_SHA

I followed the trail of calls, and it seems like the command that executes these tests is:

wp-scripts test-e2e --config packages/e2e-tests/jest.performance.config.js

which makes me believe adding the following line to jest.performance.config.js (in addition to updating workflow.yml) will do the trick:

		'<rootDir>/config/setup-debug-artifacts.js',

ellatrix · 2020-12-07T09:24:13Z

@adamziel Thanks! I think I added it correctly here? https://github.com/WordPress/gutenberg/pull/25775/files#diff-099a51d6bcb1f9eae0b587c6857bf79827750ae3dc210db79d736f65b6abc165

adamziel requested review from karmatosed and mapk as code owners November 3, 2020 16:38

adamziel requested review from ajitbohra, nerrad, ntwb and talldan as code owners November 3, 2020 17:07

adamziel changed the title ~~[Try] Store the screenshots of e2e failures~~ [Try] Store the screenshots of CI e2e failures Nov 3, 2020

gziolo added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Nov 7, 2020

adamziel changed the title ~~[Try] Store the screenshots of CI e2e failures~~ Store screenshots of CI e2e failures as CI artifacts Nov 10, 2020

adamziel self-assigned this Nov 10, 2020

adamziel force-pushed the add/e2e-failures-screenshots branch from 11a7597 to 21f4d35 Compare November 10, 2020 21:24

noahtallen reviewed Nov 10, 2020

View reviewed changes

packages/e2e-tests/config/setup-screenshots.js Outdated Show resolved Hide resolved

noahtallen approved these changes Nov 10, 2020

View reviewed changes

packages/e2e-tests/config/setup-screenshots.js Outdated Show resolved Hide resolved

packages/e2e-tests/config/setup-screenshots.js Outdated Show resolved Hide resolved

noahtallen reviewed Nov 10, 2020

View reviewed changes

adamziel force-pushed the add/e2e-failures-screenshots branch from 21f4d35 to c9d9da5 Compare November 12, 2020 13:28

adamziel force-pushed the add/e2e-failures-screenshots branch from c9d9da5 to 42c60da Compare November 12, 2020 13:44

squash

d0e4737

adamziel force-pushed the add/e2e-failures-screenshots branch from 42c60da to d0e4737 Compare November 12, 2020 13:44

adamziel added 4 commits November 12, 2020 14:45

Remove artifacts

691d140

Rename screenshots to artifacts

0725e4d

remove usages

2d45bcd

fix broken test

5238318

adamziel merged commit 981b3c8 into master Nov 12, 2020

adamziel deleted the add/e2e-failures-screenshots branch November 12, 2020 17:20

github-actions bot added this to the Gutenberg 9.4 milestone Nov 12, 2020

gziolo reviewed Nov 12, 2020

View reviewed changes

packages/e2e-tests/config/setup-debug-artifacts.js Show resolved Hide resolved

gziolo reviewed Nov 12, 2020

View reviewed changes

packages/e2e-tests/config/setup-debug-artifacts.js Show resolved Hide resolved

gziolo reviewed Nov 12, 2020

View reviewed changes

packages/e2e-tests/config/setup-debug-artifacts.js Show resolved Hide resolved

This was referenced Nov 13, 2020

[e2e screenshots] Janitorial PR - add /artifacts to .gitignore and use process.cwd() instead of __dirname #26957

Merged

[try] Record failed e2e tests as artifacts #26959

Closed

kevin940726 mentioned this pull request Jan 24, 2021

Fix e2e test failure screenshots not capturing at the right time #28449

Merged

6 tasks

swissspidy mentioned this pull request May 17, 2021

E2E Tests: take screenshots on failures GoogleForCreators/web-stories-wp#7550

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store screenshots of CI e2e failures as CI artifacts #26664

Store screenshots of CI e2e failures as CI artifacts #26664

adamziel commented Nov 3, 2020 •

edited

Loading

github-actions bot commented Nov 3, 2020 •

edited

Loading

talldan commented Nov 5, 2020 •

edited

Loading

adamziel commented Nov 5, 2020

adamziel commented Nov 10, 2020

noahtallen left a comment

noahtallen Nov 10, 2020

adamziel Nov 12, 2020 •

edited

Loading

adamziel Nov 12, 2020 •

edited

Loading

testerez Nov 12, 2020

noisysocks commented Nov 11, 2020

jasmussen commented Nov 11, 2020

adamziel commented Nov 12, 2020 •

edited

Loading

gziolo commented Nov 12, 2020

ellatrix commented Dec 7, 2020

adamziel commented Dec 7, 2020

ellatrix commented Dec 7, 2020

Store screenshots of CI e2e failures as CI artifacts #26664

Store screenshots of CI e2e failures as CI artifacts #26664

Conversation

adamziel commented Nov 3, 2020 • edited Loading

github-actions bot commented Nov 3, 2020 • edited Loading

talldan commented Nov 5, 2020 • edited Loading

adamziel commented Nov 5, 2020

adamziel commented Nov 10, 2020

noahtallen left a comment

Choose a reason for hiding this comment

noahtallen Nov 10, 2020

Choose a reason for hiding this comment

adamziel Nov 12, 2020 • edited Loading

Choose a reason for hiding this comment

adamziel Nov 12, 2020 • edited Loading

Choose a reason for hiding this comment

testerez Nov 12, 2020

Choose a reason for hiding this comment

noisysocks commented Nov 11, 2020

jasmussen commented Nov 11, 2020

adamziel commented Nov 12, 2020 • edited Loading

gziolo commented Nov 12, 2020

ellatrix commented Dec 7, 2020

adamziel commented Dec 7, 2020

ellatrix commented Dec 7, 2020

adamziel commented Nov 3, 2020 •

edited

Loading

github-actions bot commented Nov 3, 2020 •

edited

Loading

talldan commented Nov 5, 2020 •

edited

Loading

adamziel Nov 12, 2020 •

edited

Loading

adamziel Nov 12, 2020 •

edited

Loading

adamziel commented Nov 12, 2020 •

edited

Loading