Experimental PR: Research spike - Cypress + Accessibility testing

[machikoyasuda]

🚧 🚧 🚧 This is an experimental research spike PR 🚧 🚧 🚧

What this PR does

• Adds 1 new spec, accessibility.cy.js that goes through 7 pages (home, agency home, eligibility, elig start, enrollment index, enrollment success, help) of the app and runs each of the following tests on each page.
• Adds several Cypress plugins that test pages for accessibility:

1. Axe - https://www.deque.com/axe/ via https://github.com/component-driven/cypress-axe
2. Pa11y -https://pa11y.org/ via https://github.com/mfrachet/cypress-audit
3. Google Lighthouse - https://developer.chrome.com/docs/lighthouse/overview/ via https://github.com/mfrachet/cypress-audit

• Read on for research results

Comparing the tools locally

Initial reading

• https://www.craigabbott.co.uk/blog/combining-axe-core-and-pa11y/
• https://www.craigabbott.co.uk/blog/axe-core-vs-pa11y/
• His conclusion: So, I stand by my original conclusion, that we should use both PA11Y and axe-core. But it looks like the implementation of using both is actually so easy that you'd be a fool not to.

Testing method

• Install pa11y npm command line tool: https://github.com/pa11y/pa11y
• Install Axe Firefox extension: https://www.deque.com/axe/devtools/firefox-browser-extension/
• Run pa11y on the command line on all 7 pages for pages on localhost (doesn't have reCaptcha, has dev debug bar) and test (has reCaptcha).
• Run Axe Firefox extension on all 7 pages for pages on localhost (doesn't have reCaptcha, has dev debug bar) and test (has reCaptcha).
• Run Google Lighthouse testing for accessibility, best practices and performance.

Results:

• Running pa11y on the command line is very fast. Returns results like this:

pa11y https://test-benefits.calitp.org/                   

Welcome to Pa11y

 > Running Pa11y on URL https://test-benefits.calitp.org/

Results for URL: https://test-benefits.calitp.org/

 • Error: This element has insufficient contrast at this conformance level. Expected a contrast ratio of at least 4.5:1, but text in this element has a contrast ratio of 4.2:1. Recommendation:  change text colour to #005681.
   ├── WCAG2AA.Principle1.Guideline1_4.1_4_3.G18.Fail
   ├── #skip-to-content > div
   └── <div class="container">Skip to Main Content</div>

1 Error

• Axe looks like this:

• Axe run on a page with an open modal:

Preliminary findings

Like the blog posts above asserted, the tools find different errors. I was impressed that pa11y was able to find the Skip Nav color constrast bug, even though I never told the tool to go focus on it or click tab. The Axe tool didn't find the Skip Nav color contrast bug. But it did find "Best Practice" bugs that pa11y doesn't have, like how all page content is contained by landmarks. I came away thinking we should be using both tools in development - b/c both tools - command line and Firefox / Chrome extension are quite fast -- much faster than running a full Google Lighthouse test. (Lighthouse performance testing adds a lot of time.) Also, if the tools are easy enough to integrate into Cypress and reliable, it's worth while to add them both to the CI test suite as well.

Axe flags
Lighthouse, meanwhile, returns a score of 100 for accessibility - even though both pa11y and Axe found error level issues. So it might not be worth it to run Lighthouse for accessibility in addition to the 2 other tools.

Lastly, it's pretty helpful that the pa11y error message suggested an alternate color for the insufficient contrast. #005681 is really close to what we have.

Comparing the tools in Cypress

Installing and configuring tooling

• Both https://github.com/component-driven/cypress-axe and https://github.com/mfrachet/cypress-audit were easy to install and test. Cypress-axe has 3x greater usage numbers.

Testing

• How it works:

1. Write all the code necessary to get to the page you want to test in the before()
2. Then add the following:

      cy.pa11y(pa11yOpts);
      cy.lighthouse(lighthouseOpts);
      cy.checkA11y(null, {
        includedImpacts: ["critical"],
      });

3. The options objects are all optional. I added these options to reflect what is currently failing and won't pass the test. Removing these options will cause the tests to fail.

• Pa11y fails for color contrast
• Checka11y (axe) fails for various aria landmarks missing.
• Lighthouse is currently set to only test performance, accessibility and best practices, so it excludes SEO and PWA (Progressive Web App, b/c our app isn't one). Performance is a bit hard to test with Cypress b/c it's not testing a production bundle (https://mfrachet.github.io/cypress-audit/guides/lighthouse/good-to-know.html#test-with-a-production-bundle).

Lighthouse issues

Performance test is flakey and returns different results after repeated runs.

Pa11y issues

The Pa11y color contrast rule seems to be catching more things than I think it should.

1. 

This is flagging the H1 and H2 as insufficient contrast, but I think it's because the image in the background isn't showing up.

2. 

This is flagging the Previous Page button as insufficent contrast. But even the pa11y command line tool doesn't find this or flag this - it only flags the Skip Nav. So we're getting different errors depending on whether running in the CLI or Cypress. Perhaps this will all disappear if we just change the Skip Nav color.

% pa11y https://test-benefits.calitp.org/eligibility/start

Welcome to Pa11y

 > Running Pa11y on URL https://test-benefits.calitp.org/eligibility/start

Results for URL: https://test-benefits.calitp.org/eligibility/start

 • Error: This element has insufficient contrast at this conformance level. Expected a contrast ratio of at least 4.5:1, but text in this element has a contrast ratio of 4.2:1. Recommendation:  change text colour to #005681.
   ├── WCAG2AA.Principle1.Guideline1_4.1_4_3.G18.Fail
   ├── #skip-to-content > div
   └── <div class="container">Skip to Main Content</div>

1 Errors

The Pa11y library is supposed to allow you to ignore certain rules. I tried adding an ignore: for the color contrast rule here https://github.com/cal-itp/benefits/compare/spike/cypress-a11y?expand=1#diff-8bcefc461aeb1ada5582435f5b2427508a858457bbc046a7a968dc0e1b76c222R8, but I don't think it's working properly in the test runner. It works fine locally for me though:

% pa11y --ignore "WCAG2AA.Principle1.Guideline1_4.1_4_3.G18.Fail" localhost:8000

Welcome to Pa11y

 > Running Pa11y on URL http://localhost:8000

No issues found!

So that's why I set the threshold to a higher number of 4 to allow the tests to pass.

Evaluation

• Test the performance: Figure out how many more seconds this adds to the CI test suite
• Test the accuracy: Ensure the test results on Cypress locally are the same as that of running the tests on Cypress on GitHub Actions. Ensure the tests fail and then pass on Cypress on GitHub Actions.
• Test the accuracy: Make the necessary code tests, then run the tests locally and on GitHub Actions. The tests should now pass without the option flags.
• Test the performance of multiple tests: Try running the tests on multiple PRs to ensure tests aren't brittle/returning inconsistent results.

Conclusion

For Cypress, I think we should keep axe but ditch cypress-audit (pa11y and lighthouse). Though I appreciate that the pa11y tool found the Skip Nav bug, the color contrast rule on pa11y has too many false positives when running in Cypress. We can still use the tool locally during the development of new components with new color combinations. Color contrast is also being checked in Figma at earlier parts of the design phase. Since the axe testing is more sensitive than lighthouse for accessibility, I don't think we need to add any more new lighthouse testing. I think keeping the Lighthouse tests as it is, for the home page, help page, is sufficient for testing performance and best practices. For the other pages with forms, the axe tool is a much more thorough test than Lighthouse.

What's next

Possible next steps:

• Add GitHub issues for all the Axe, pa11y issues.
• Resolve the Axe and pa11y issues. Developers might want to add these tools to their local machine, browsers to familiarize themselves with all these rules where the app is failing.
• Then add the test suite to the CI, if we evaluate that it's worth it.
• Write more specific test cases: like opening every single modal and running the tests on a page with an open modal, running the tests in mobile/small window width.

[machikoyasuda]

The latest: Tests run fine locally, but return different results on GitHub Actions. Namely, they always pass when they should not be! I tried adding an explicit support file, adding logging, re-writing the entire Action, but none of those fixed it. Not really sure what the issue is. Also didn't work: Running in headed mode (as opposed to headless Chrome).

[machikoyasuda]

Experiment is over! Closing + full write-up will be out by end of week.