E2E Integration Tests

[pavish]

Currently we have the setup for unit tests in backend and frontend.

This thread is to decide upon End-to-end integration tests and when to proceed with it (currently, at the date of start of this discussion, it is too early to set up).

My recommendation is to use Cypress, and create test cases mimicking user behaviour.

We can brainstorm other options/thoughts around this.

[mathemancer]

What's the advantage of Cypress over Selenium? I ask because the comparisons I found seemed to give the edge to Cypress in speed, but that's because it doesn't use browser drivers. Wouldn't the browser driver approach make the tests more useful and granular? I.e., we could notice if something is working for all browsers except Safari and change things appropriately.

[mathemancer]

TBH, I'm convinced by your points there to give Cypress a shot (especially now that Edge is Chromium-based)

I definitely like the open-source CI/CD integration

[kgodey]

@pavish Could you elaborate more on the CI/CD integration tools for Cypress? I've run Selenium tests in CI before, I didn't need any special tooling to do it.

I don't have a strong opinion on Selenium vs. Cypress, I'd just like more details on this point.

[kgodey]

We might also want to check out Playwright (https://github.com/microsoft/playwright, https://github.com/microsoft/playwright-test), which does have WebKit support. I think it's important to test all major browsers.

Here's a Hacker News discussion about Cypress covering a lot of cons, which might be useful: https://news.ycombinator.com/item?id=25381242

[kgodey]

Here's another good article: https://www.testim.io/blog/puppeteer-selenium-playwright-cypress-how-to-choose/

[pavish]

@kgodey This is from Cypress docs on CI/CD integration: https://docs.cypress.io/guides/continuous-integration/github-actions. I have not used Selenium on CI before, and I couldn't find a good documentation on it other than some SAAS services.

I do not have a strong preference towards Cypress, I personally found it more easier to use than Selenium. I guess we'll have to look more into the pros and cons of both. I am open to any solution that is the most optimal for us.

[pavish]

Based on our requirements, browser support and ease of CI/CD integration, I'm currently inclined to go with Playwright, although we'd have to integrate it and test it's viability.

@kgodey @mathemancer Please let me know what you think.

[kgodey]

I'm fine with going with Playwright, I think it's best to just pick an option and test it rather than spend a lot of time on research. It looks like it supports all the major browsers, which was my main concern.

[mathemancer]

I'll defer to your decision here.

[zackkrida]

I can 👍 the sentiment from that Hacker News thread about Cypress. It's very powerful, and the GUI when testing locally is really, really nice. The test output is very legible, and I really want to like it...but, the API for writing tests is definitely a little frustrating.

Something like Playwright (which is basically just cross-browser Puppeteer) is great, and now has a built-in test runner that makes it even easier to work with. https://playwright.dev/docs/test-runners

Example playwright test:

import { test, expect } from '@playwright/test';

test('basic test', async ({ page }) => {
  await page.goto('https://playwright.dev/');
  const name = await page.innerText('.navbar__title');
  expect(name).toBe('Playwright');
});

[kgodey]

Thanks @zackkrida!