Fix issues that caused gamescope to abort at exit

[sharkautarch]

The first commit fixes issue #1305
Note: I used a user event instead of the sdl quit event, to ensure that the sdl thread is only closed from inside steamcompmgr_exit()

After fixing the aforementioned issue, I noticed that gamescope would still sometimes abort/coredump at exit.

So I found a separate issue, wherein the present_wait thread, when running GetVBlankTimer().MarkVBlank( vblanktime, true );, would segfault upon trying to access a backend while steamcompmgr_exit() was deleting said backend.
The second commit fixes this w/ an awkward dance between the two threads, where steamcompmgr_exit() asks the present_wait thread to exit, and then waits for the present_wait thread to respond back.
The reason why I did that weird stuff in the second commit is that I wanted to minimize the overhead being added to present_wait_thread_func(): for most of the runtime, this commit will not add any delay to present_wait_thread_func()

update: Slight edit to the second commit: in steamcompmgr_exit(), I moved the atomic store to g_presentThreadShouldExit so that it happens before the atomic store to g_currentPresentWaitId, just to cover any rare edge case w/ atomic load/store timing

[sharkautarch]

Last Update: for some reason, the present thread needs to run g_presentThreadShouldExit.notify_all() when exiting, to ensure that steamcompmgr_exit() doesn't wait forever. So I added that in

[matte-schwartz]

just as a heads up @sharkautarch this PR causes both a gamescope and an xwayland coredump once the "Switch to Desktop" shortcut is used within gamescope-session on my 7900XTX on Fedora 40. this results in a 20-40 second freeze before the session switch executes properly and returns me to desktop.

gamescope-coredump.log
xwayland-coredump.log
gamescope-stdout-xwayland-coredump.log

edit: also, the same issue happens when applied against upstream gamescope master so you can disregard the gamescope-plus logging in gamescope-stdout

[sharkautarch]

@matte-schwartz
Hmm so something is causing Xwayland to abort…
First, make sure you have asan and ubsan installed:
Make sure you have the subpackages of gcc: libasan, libubsan installed

Then, try running gamescope with these environment variables to run w/ address sanitizer:
ASAN_OPTIONS= dump_instruction_bytes=1:halt_on_error=0:fast_unwind_on_malloc=0:detect_leaks=0:leak_check_at_exit=0:handle_abort=1:intercept_tls_get_addr:1 LD_PRELOAD="$(gcc -print-file-name=libasan.so) $(gcc -print-file-name=libubsan.so)"

Note that when running w/ address sanitizer, coredumping is disabled, because otherwise the coredumps would be terabytes in size…

I’m not sure if you might need to actually recompile gamescope w/ address sanitizer instrumentation in order to catch whatever issue is going on…
Here’s how to compile with address sanitize:
just add these flags to meson setup:
-Db_sanitize=address,undefined -Dc_args="-fsanitize-recover=all -w -Wno-error=calloc-transposed-args -Wno-error=stringop-overflow" -Dc_link_args="-fsanitize-recover=all -w -Wno-error=calloc-transposed-args -Wno-error=stringop-overflow" -Dcpp_args="-fsanitize-recover=all -w -Wno-error=calloc-transposed-args -Wno-error=stringop-overflow" -Dcpp_link_args="-fsanitize-recover=all -w -Wno-error=calloc-transposed-args -Wno-error=stringop-overflow" --buildtype=debugoptimized

[matte-schwartz]

Then, try running gamescope with these environment variables to run w/ address sanitizer: ASAN_OPTIONS= dump_instruction_bytes=1:halt_on_error=0:fast_unwind_on_malloc=0:detect_leaks=0:leak_check_at_exit=0:handle_abort=1:intercept_tls_get_addr:1 LD_PRELOAD="$(gcc -print-file-name=libasan.so) $(gcc -print-file-name=libubsan.so)"

I gave this a try but keep getting an error about needing to link asan at runtime, even though I built gamescope following your instructions below. Let me read up a bit on how these tools are supposed to work so I can try and figure out what's going wrong.

I’m not sure if you might need to actually recompile gamescope w/ address sanitizer instrumentation in order to catch whatever issue is going on… Here’s how to compile with address sanitize: just add these flags to meson setup: -Db_sanitize=address,undefined -Dc_args="-fsanitize-recover=all -w -Wno-error=calloc-transposed-args -Wno-error=stringop-overflow" -Dc_link_args="-fsanitize-recover=all -w -Wno-error=calloc-transposed-args -Wno-error=stringop-overflow" -Dcpp_args="-fsanitize-recover=all -w -Wno-error=calloc-transposed-args -Wno-error=stringop-overflow" -Dcpp_link_args="-fsanitize-recover=all -w -Wno-error=calloc-transposed-args -Wno-error=stringop-overflow" --buildtype=debugoptimized

[sharkautarch]

@matte-schwartz
Is the message saying something like: ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD ?

If so, try running ASAN again, but this time with the verify_asan_link_order=0 option:
ASAN_OPTIONS=verify_asan_link_order=0:dump_instruction_bytes=1:halt_on_error=0:fast_unwind_on_malloc=0:detect_leaks=0:leak_check_at_exit=0:handle_abort=1:intercept_tls_get_addr:1 LD_PRELOAD="$(gcc -print-file-name=libasan.so) $(gcc -print-file-name=libubsan.so)"