GitHub Actions: Run accessibility checks on additional pages

[afeld]

We currently run Lighthouse on a couple pages:

benefits/.github/workflows/tests-ui.yml

Lines 44 to 49 in f2503e5

	- name: Run Lighthouse tests for a11y
	uses: treosh/[email protected]
	with:
	urls: |
	http://localhost:8000
	http://localhost:8000/help

It would be helpful to have it (or some other automated accessibility checker run against other pages as well.

Acceptance Criteria

• Get automated reports of accessibility violations on [at least the majority of] pages in Benefits

Additional context

Some options:

• https://www.npmjs.com/package/cypress-lighthouse
• https://mfrachet.github.io/cypress-audit/
• https://github.com/component-driven/cypress-axe#readme

[indexing]

@machikoyasuda The other day you mentioned we are running Lighthouse checks, so I believe we can close this issue?

[machikoyasuda]

@indexing This issue is referring to adding automation to run Lighthouse checks on every page on every PR. So far, we're only running it on the home page and the help page. Not sure if this Issue is a priority anymore though.

[machikoyasuda]

@indexing To me, this seems like something that could be a 1-week stretch ""fun"" ticket for a developer to work on during a slush week, if there aren't any other similar tickets. It's definitely not a must have.

[indexing]

@thekaveman What are you thoughts? Is it worth expanding the views we're checking via Lighthouse?

[thekaveman]

I agree with @machikoyasuda this seems like a good exercise to timebox for a Stretch week.

Generally it would be good to do more automated checking like this (I rarely if ever remember to run Lighthouse myself locally). Let's see what we can accomplish in a reasonable amount of time using just Cypress and one of the plugins above, but without worrying about mocking all the external services or getting through e.g. Login.gov just yet.

[machikoyasuda]

Beginning preliminary research on this. May continue it into next week/stretch week.

[machikoyasuda]

After extensive testing #1675 last week, I've come to the conclusion that, at this time, we shouldn't add the cypress-axe or cypress-audit tests to our continuous integration GitHub Actions. The tests worked perfectly well locally, but resulted in false positives when run on GitHub Actions.

Steps we can take now:

• Fix the current known accessibility errors:

1. Skip nav color contrast - Affects every page.
2. Headline hierarchy bug (Refactor templates and includes #1528 bug reported here) for pages that use a Media Item, but don't have a H2. - Affects 1 page.
3. Aria labeling for Modals - Affects all pages with modals.

• Internal training: Start using the browser extensions for Axe (Firefox, Chrome), pa11y command line tool https://pa11y.org/ for faster, more reliable local testing. Prefer these 2 tools to Google Lighthouse. Use these tools when developing any new user interfaces.

• Once all the existing accessibility issues are fixed, re-try the Cypress a11y testing on GitHub Actions. Maybe there will be improvements to the false positives. If Cypress continues to be inconsistent, consider moving to Playwright (https://playwright.dev/, https://playwright.dev/docs/accessibility-testing) - which comes with Docker-support and GitHub Action-support of out of the box (Re-add option for running Cypress in a container #681) (https://playwright.dev/docs/docker)

Either way, we have to remember that automated testing only provides so much reassurance - and we'll always need some manual testing. And also, that certain accessibility issues (like color contrast) should be detected earlier at the design level.

As an example, both of the aforementioned tools weren't able to catch that a button had insufficient color contrast - because aria was set to disabled for that button! When I took aria-disabled=true out from the button, both tools were able to notice the button's issues.

cc @thekaveman @angela-tran

[thekaveman]

@machikoyasuda what do you think about closing this issue now that you've done the research and made recommendations?