Model based testing made easy with xstate-marionettist

[rjdestigter]

I've taken some boiler plate code that I have been using in projects at work and published it into a package!: https://github.com/rjdestigter/xstate-marionettist

I'm using this to make writing model based tests using Jest, XState, and Puppeteer quicker and easier. Especially if you want to write end-to-end (ish) tests that mock all possible network responses. You can try it out and have a look by cloning the project or follow the instructions on how to add it to your project. I would really love some feedback on this.

The goal is really to just have a simple "language" to create these tests and not have to worry about integrating XState with Puppeteer.

[tomByrer]

Your API seems neat!

Thought about using Playwright? It has similar API Puppeteer; same old devs even! But Playwright from start aimed to be cross-browser compatible, vs Puppeteer had been Chrome-focused. Has been a while since I looked at them though.

[rjdestigter]

I haven't but I have heard of them yes! I should have a look. If the API is quite similar then it would be fairly easy to support both.

[mattpocock]

This looks great! I am more pro-Cypress than pro-puppeteer. Is there any way you could pass in an API via a prop and have Typescript infer which one you chose?

Or are there things specific to puppeteer you need to make this work?

[rjdestigter]

I have been thinking about this yes. The request interception is very Puppeteer specific at the moment but I have experimented with Cypress as well in the past.

[rjdestigter]

This whole thing has primarily been born out of a need to intercept and mock all network requests. The project at work didn't have good support for full end-to-end tests. But also all network responses are strictly typed and validated with io-ts so in a sense all possible responses (including failures) were known and could confidently be mocked.

The tricky part is looking at the generated test model's pathway and deducing what events are going to cause network requests that should be interrupted.

[rjdestigter]

I've got another example here https://github.com/rjdestigter/xstate-marionettist-conduit that is testing https://realworld.svelte.dev/ It was a really good exercise writing a test for something I didn't build. I found a few bugs and edge cases that need fixing.

The difficult part is understanding when to defer network requests and when to continue resolving them. I would like to make that more inuitive.

[rjdestigter]

I've added support for Playwright as well.

[ezhikov]

That's awesome. I had multiple attempts to start using @xstate/test, but I couldn't make it work with network stubbing, since in first state there is multiple requests with different responses each. I'll give it a try today (though, looks like I have to migrate from cypress to playwright or puppeteer)

[rjdestigter]

Cool! Let me know how it goes. I do want to try adding Cypress support as well since I did at some point figure out / hack a similar solution with it.

[ezhikov]

So, I failed to test anything apart from initial page state where loader present and redirect to login page on auth check error, but I don't think that the problem is in your library. Our first request is in iframe that should post message into main window and, for some reason, chromium decided that there is some mismatch. I also had to do small monkey patching to make marionettist accept content-type different from application/json.

I also was confused about beforVisit configuration field. I saw it in your conduit example and, of course tried to use beforeVisit instead. Not sure id this is intentional or a typo.

Also, when I deferred some requests before visit they were never resolved.

Now, to the good parts. When I understood how it works I managed to cover first part of the application (with all the different outcomes) pretty quickly. I'll definitely will try again when I'll have more time.

[rjdestigter]

Awesome, thanks for checking it out and the feedback!

beforVisit

Definitely a typo!

I also had to do small monkey patching to make marionettist accept content-type different from application/json

Yeah, the state of the code is highly derived from my work project still. I'm currently fighting with Cypress. Once I get that down I'm going to split the code up into core, puppeteer, playwright, and a cypress package and start working on better abstraction and configuration.

Also, when I deferred some requests before visit they were never resolved.
Did you use the resolve action after the visit?

[ezhikov]

Definitely a typo!

Wasn't sure, since you use it in code itself and in conduit example.

cypress package

Can't wait to try it out

[rjdestigter]

I've got some preliminary support for Cypress done! https://github.com/rjdestigter/xstate-marionettist/tree/cypress/packages/cypress

I haven't published anything yet because I have to update the README and I want to add some more (edge) cases and make sure everything is stable. I am using the experimental cy.route2 API to intercept network requests.

The code has cleaned up quite a bit and shareable abstractions between all three test runners is located in a core folder.

Running yarn in the root will also install packages for the example app. If you want to check it out run:

yarn
yarn build
cd example/app
yarn start && yarn cypress open

[ezhikov]

Looks interesting. Also, fixtures should work out of the box, if it accepts '@fixture' in reply method.

Do you mind if I make PR to also accept method and custom response headers (at least Content-Type) next week?

[rjdestigter]

Not at all!

[rjdestigter]

Alright, it's all out the door. Cypress' route2 API is definitely not fully stable yet though. To make it work with the conduit example I had to add an additional cy.wait(50) between each path iteration allowing Cypress to do some network related cleanup before the next test.

You can experiment with it by installing xstate-marionettist-cypress and add:

{
  "experimentalNetworkStubbing": true,
}

to your cypress.json configuration.

[ezhikov]

I just finished playing with example app. Couldn't make fixtures work, but otherwise it's awesome. Apparently, we have either use req.reply callback or fixtures and I need more time to dig deeper into code to figure out how to combine them, since most of useful data extracted in callback.