ci: Set the "render" kind on tests that need renderer, mark them as heavy for nextest

[torokati44]

But it doesn't seem to work locally for me...? 😵‍💫

Turns out, updating cargo-nextest fixed it for me...

Relevant docs pages:
https://nexte.st/docs/configuration/threads-required/
https://nexte.st/docs/filtersets/reference/

This will hopefully stop random crashes of visual tests (especially on avm2/pixelbender_shaderdata and avm2/graphics_round_rects) on GHA.

[Dinnerbone]

I think I'd prefer a suffix, usually things are module::path::testname, so this kinda fits

[torokati44]

Yeah, that's what I tried first, but...

It kinda makes render look like it's the test name, and the test name like it's something else...?

[Dinnerbone]

Hmmm okay. Prefix looks fine then

[torokati44]

Cool! Does this count as a green checkmark? 🤔

[kjarosh]

Unfortunately this will not work properly. There is a certain contract we have to adhere to when using libtest_mimic — if we list a test, we should also be able to find it by its name. In case of nextest, tests which do not comply with this requirement will not raise any errors and will be skipped and marked as passed.

Compare logs from this PR and from master. The test visual/shumway_acid_tests/acid_big took ~0.005s instead of ~1.889s.

I would personally abstain from playing with test names (it's very easy to skip some tests by accident that way), and try using test kinds for that purpose, as we already have support for them (and tests with a kind are being looked up properly).

[torokati44]

Good catch, thanks!

Can a test have any number of kinds though?

[kjarosh]

Can a test have any number of kinds though?

A test may have one kind only, but we can join multiple kinds with a comma (I think, not entirely sure it'll work) and add support for multiple kinds that way ourselves.

[kjarosh]

Now thinking about it... a test kind is just a string prefix (but inside brackets), which is ignored when looking up tests. I think we can safely put multiple strings separated by a comma there and it'll work, even if we pass no kind or an invalid kind when looking up tests.

The only situation when the contents of a kind matter is when we parse them, so in .config/nextest.toml and possibly in the code.

[torokati44]

Okay, stole some bits and pieces from #15950, now render is a kind, not a name prefix.

[kjarosh]

I believe the nextest's kind is something different from the libtest_mimic's kind. See https://nexte.st/docs/filtersets/reference/#binary-kinds

[kjarosh]

I think you'd have to use a regular expression with test(...), e.g. test(/\[render\] .*/)

[kjarosh]

Or rather: test(/^\[render\] /)

[torokati44]

How about a glob pattern? like test(#[render] *). Less escaping!
EDIT: Hmmm, it doesn't work...

[torokati44]

Aaaand now it automatically does the [render] tests in one group first, so it makes the failure more likely... Let's try a weight of 3, so no two render tests can run at the same time, but other ones can together with them...?

[torokati44]

Even one at a time they crash... 😨😵😵‍💫🫥🫠

[kjarosh]

Maybe just set a higher number of retries for render tests for now?

[torokati44]

Maybe just set a higher number of retries for render tests for now?

Interestingly, it's again only the same two tests that crashed. So I'd allow retries only for those then - but then we're almost back to square one.

[torokati44]

Finally, a stack trace!

https://github.com/ruffle-rs/ruffle/actions/runs/10488227410/job/29050242404?pr=17574#step:8:7584

Thread 11 (Thread 0x7fffc7e006c0 (LWP 14650) "tests-6cab02727"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c4526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c297b6 in __libc_message_impl (fmt=fmt@entry=0x7ffff7dce8d7 "%s\n") at ../sysdeps/posix/libc_fatal.c:132
#6  0x00007ffff7ca8fe5 in malloc_printerr (str=str@entry=0x7ffff7dd1ae0 "double free or corruption (!prev)") at ./malloc/malloc.c:5772
#7  0x00007ffff7cab11c in _int_free_merge_chunk (av=0x7ffff7e03ac0 <main_arena>, p=0x5555575a0020, size=1040) at ./malloc/malloc.c:4679
#8  0x00007ffff7cab42a in _int_free (av=0x7ffff7e03ac0 <main_arena>, p=<optimized out>, have_lock=<optimized out>) at ./malloc/malloc.c:4646
#9  0x00007ffff7cadd9e in __GI___libc_free (mem=0x5555575a0030) at ./malloc/malloc.c:3398
#10 0x00007fffeee63f17 in void std::vector<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> >, std::allocator<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> > > >::_M_realloc_insert<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> > >(__gnu_cxx::__normal_iterator<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> >*, std::vector<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> >, std::allocator<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> > > > >, std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> >&&) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#11 0x00007fffeee63443 in llvm::PassRegistry::registerPass(llvm::PassInfo const&, bool) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#12 0x00007fffeef94ec7 in ?? () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#13 0x00007ffff7ca1ec3 in __pthread_once_slow (once_control=0x7ffff55fc194, init_routine=0x7ffff64e93b0 <__once_proxy>) at ./nptl/pthread_once.c:116
#14 0x00007fffeef94ded in llvm::initializeEarlyIfConverterPass(llvm::PassRegistry&) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#15 0x00007fffeef40042 in llvm::initializeCodeGen(llvm::PassRegistry&) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#16 0x00007fffef2c3dec in llvm::TargetPassConfig::TargetPassConfig(llvm::LLVMTargetMachine&, llvm::legacy::PassManagerBase&) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#17 0x00007ffff1a1727f in ?? () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#18 0x00007fffef030d5a in llvm::LLVMTargetMachine::addPassesToEmitMC(llvm::legacy::PassManagerBase&, llvm::MCContext*&, llvm::raw_pwrite_stream&, bool) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#19 0x00007ffff0837ac9 in llvm::MCJIT::emitObject(llvm::Module*) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#20 0x00007ffff0837c6d in llvm::MCJIT::generateCodeForModule(llvm::Module*) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#21 0x00007ffff08386ed in llvm::MCJIT::finalizeObject() () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#22 0x00007ffff07bf287 in LLVMGetPointerToGlobal () from /lib/x86_64-linux-gnu/libLLVM-17.so.1

"double free or corruption (!prev)")

Hmm, LLVM bug?

[torokati44]

Cross-referencing, about the crash: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11797