haskell-modules/generic-builder.nix: Allow running Haskell tests as passthru.tests

[TeofilC]

Description of changes

This adds a flag to the Haskell generic builder to allow running each test-suite of a package in a distinct passthru.tests._ derivation.

This is quite helpful for large/complex test suites attach to large libraries. Having these as separate derivations makes it a lot quicker to debug why a test suite is failing. It's also quite nice for end-users cause there's a clearer separation between "this build failed" and "this test-suite failed".

If a test requires KVM, this is also helpful for just making the test derivation require that.

This is a feature in haskell.nix and I'm taking inspiration from their implementation. (I'll add credit in the commit message once I write up a proper one). For reference here's the equivalent bit of haskell.nix: https://github.com/input-output-hk/haskell.nix/blob/master/lib/check.nix, which has a few more bells and whistles.

This is dependent on NixOS/cabal2nix#617 to get the list of test suites at eval time.

I'm really uncertain about the name for the option. If anyone has ideas I'd appreciate it!

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[maralorn]

I like the idea of this, but I think we need a bit more discussion on the design.

I am not certain whether this design as is will make anything easier. Concretely can you name a kind of iteration on the tests which will not invalidate building the original package?

[TeofilC]

I am not certain whether this design as is will make anything easier. Concretely can you name a kind of iteration on the tests which will not invalidate building the original package?

@maralorn Thanks for taking a look! That's a good point. By default they will have the same source and whenever the main build rebuilds, this will also rebuild.
If the tests are fully defined in Haskell code then there's no way around this. But for golden test style tests, you can exclude the golden files from the build for the test exe, and then include it in the build for the test runner derivation. This would mean that the runner derivation can be updated with the new golden files without the exe derivation having to rebuild.

I also find that tests need more maintenance than regular libraries. They tend to require more of an environment such as extra exes, resources (eg, the time zone database), or other services. This means they tend to need a lot of small tweaks and it's nice to only have to rebuild the test derivation when that happens. Also when you have other serivces it's quite nice to be able to have this as a separate derivation cause then you can turn it into a VM based build and have things set up in a more principled way. And means your main derivation won't rebuild when these potentially loosely coupled things change.

The main advantage of this approach though isn't in the relationship between an individual main derivation and the test deriviation, but in its effects on the overall build graph of a larger project. As it stands downstream libraries will not be built if the test suite fails. This is quite annoying if your builds are slow and leads to slow feedback loops when using the nixpkgs infrastructure to build things in CI. You end up fixing a test suite, only to find that something else downstream in the build graph was also broken but hadn't been tried. Making it so that as much of the build graph is attempted to be built even if they fail means you can cut out a lot of waiting.