Skip to content

UI visual regression testing to cover UI widgets visibility #13

UI visual regression testing to cover UI widgets visibility

UI visual regression testing to cover UI widgets visibility #13

Triggered via pull request September 19, 2024 14:39
Status Failure
Total duration 4m 40s
Billable time 5m
Artifacts 2

ui_notebooks_test.yaml

on: pull_request
verify-0_basic_ray
4m 24s
verify-0_basic_ray
Fit to window
Zoom out
Zoom in

Annotations

10 errors and 1 notice
tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs: ui-tests/tests/0_basic_ray.test.ts#L58
1) tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs Error: Screenshot comparison failed: Expected an image 656px by 86px, received 656px by 2131px. 936533 pixels (ratio 0.67 of all image pixels) are different. Expected: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts-snapshots/widgets-cell-3-linux.png Received: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-3-actual.png Diff: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-3-diff.png 56 | 57 | if (captures[i]) { > 58 | expect.soft(captures[i]).toMatchSnapshot(image); // Compare pre-existing capture | ^ 59 | continue; 60 | } 61 | } at /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts:58:34
tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs: ui-tests/tests/0_basic_ray.test.ts#L58
1) tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs Error: Screenshot comparison failed: Expected an image 592px by 78px, received 592px by 77px. 2104 pixels (ratio 0.05 of all image pixels) are different. Expected: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts-snapshots/widgets-cell-4-linux.png Received: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-4-actual.png Diff: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-4-diff.png 56 | 57 | if (captures[i]) { > 58 | expect.soft(captures[i]).toMatchSnapshot(image); // Compare pre-existing capture | ^ 59 | continue; 60 | } 61 | } at /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts:58:34
tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs: ui-tests/tests/0_basic_ray.test.ts#L58
1) tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs Error: Screenshot comparison failed: Expected an image 656px by 21px, received 656px by 159px. 2784 pixels (ratio 0.03 of all image pixels) are different. Expected: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts-snapshots/widgets-cell-5-linux.png Received: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-5-actual.png Diff: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-5-diff.png 56 | 57 | if (captures[i]) { > 58 | expect.soft(captures[i]).toMatchSnapshot(image); // Compare pre-existing capture | ^ 59 | continue; 60 | } 61 | } at /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts:58:34
tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs: ui-tests/tests/0_basic_ray.test.ts#L58
1) tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs Error: Screenshot comparison failed: 831 pixels (ratio 0.03 of all image pixels) are different. Expected: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts-snapshots/widgets-cell-6-linux.png Received: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-6-actual.png Diff: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-6-diff.png 56 | 57 | if (captures[i]) { > 58 | expect.soft(captures[i]).toMatchSnapshot(image); // Compare pre-existing capture | ^ 59 | continue; 60 | } 61 | } at /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts:58:34
tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs: ui-tests/tests/0_basic_ray.test.ts#L58
1) tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs Error: Screenshot comparison failed: Expected an image 656px by 213px, received 656px by 142px. 5620 pixels (ratio 0.05 of all image pixels) are different. Expected: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts-snapshots/widgets-cell-7-linux.png Received: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-7-actual.png Diff: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-7-diff.png 56 | 57 | if (captures[i]) { > 58 | expect.soft(captures[i]).toMatchSnapshot(image); // Compare pre-existing capture | ^ 59 | continue; 60 | } 61 | } at /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts:58:34
tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs: ui-tests/tests/0_basic_ray.test.ts#L58
1) tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs Error: Screenshot comparison failed: Expected an image 656px by 52px, received 656px by 142px. 3116 pixels (ratio 0.04 of all image pixels) are different. Expected: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts-snapshots/widgets-cell-8-linux.png Received: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-8-actual.png Diff: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-8-diff.png 56 | 57 | if (captures[i]) { > 58 | expect.soft(captures[i]).toMatchSnapshot(image); // Compare pre-existing capture | ^ 59 | continue; 60 | } 61 | } at /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts:58:34
tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs: ui-tests/tests/0_basic_ray.test.ts#L58
1) tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs Error: Screenshot comparison failed: Expected an image 656px by 213px, received 656px by 142px. 6204 pixels (ratio 0.05 of all image pixels) are different. Expected: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts-snapshots/widgets-cell-9-linux.png Received: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-9-actual.png Diff: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-9-diff.png 56 | 57 | if (captures[i]) { > 58 | expect.soft(captures[i]).toMatchSnapshot(image); // Compare pre-existing capture | ^ 59 | continue; 60 | } 61 | } at /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts:58:34
tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs: ui-tests/tests/0_basic_ray.test.ts#L58
1) tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs Error: Screenshot comparison failed: 760 pixels (ratio 0.05 of all image pixels) are different. Expected: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts-snapshots/widgets-cell-10-linux.png Received: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-10-actual.png Diff: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-10-diff.png 56 | 57 | if (captures[i]) { > 58 | expect.soft(captures[i]).toMatchSnapshot(image); // Compare pre-existing capture | ^ 59 | continue; 60 | } 61 | } at /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts:58:34
tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs: ui-tests/tests/0_basic_ray.test.ts#L58
1) tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs Error: Screenshot comparison failed: Expected an image 656px by 387px, received 656px by 142px. 10701 pixels (ratio 0.05 of all image pixels) are different. Expected: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts-snapshots/widgets-cell-11-linux.png Received: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-11-actual.png Diff: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-11-diff.png 56 | 57 | if (captures[i]) { > 58 | expect.soft(captures[i]).toMatchSnapshot(image); // Compare pre-existing capture | ^ 59 | continue; 60 | } 61 | } at /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts:58:34
tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs: ui-tests/tests/0_basic_ray.test.ts#L58
1) tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs Error: Screenshot comparison failed: Expected an image 592px by 77px, received 592px by 78px. 253 pixels (ratio 0.01 of all image pixels) are different. Expected: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts-snapshots/widgets-cell-12-linux.png Received: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-12-actual.png Diff: /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/test-results/tests-0_basic_ray-Visual-R-fdd9d-nb-and-capture-cell-outputs/widgets-cell-12-diff.png 56 | 57 | if (captures[i]) { > 58 | expect.soft(captures[i]).toMatchSnapshot(image); // Compare pre-existing capture | ^ 59 | continue; 60 | } 61 | } at /home/runner/work/codeflare-sdk/codeflare-sdk/ui-tests/tests/0_basic_ray.test.ts:58:34
🎭 Playwright Run Summary
1 failed tests/0_basic_ray.test.ts:30:7 › Visual Regression › Run notebook 0_basic_ray.ipynb and capture cell outputs

Artifacts

Produced during runtime
Name Size
ipywidgets-test-assets
1.54 MB
ipywidgets-test-report
1.73 MB