[compiler-rt][nsan] Disable coredump creation

[chestnykh]

• Disable coredump creation
  If NSAN_OPTIONS contains abort_on_error=1 the process hangs because the kernel tries to create a very huge core

msan, asan, tsan etc. runtimes disables coredumps too

Fix #98806

[chestnykh]

CC: @vitalybuka @alexander-shaposhnikov

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Dmitriy Chestnykh (chestnykh)

Changes

• Install deadly signal handlers to print an informative report if something like SIGSEGV was happened
• Disable coredump creation
• If NSAN_OPTIONS contains abort_on_error=1 the process hangs because the kernel tries to create a very huge core

Fix #98806

---

Full diff: https://github.com/llvm/llvm-project/pull/98807.diff

1 Files Affected:

• (modified) compiler-rt/lib/nsan/nsan.cpp (+17-1)

diff --git a/compiler-rt/lib/nsan/nsan.cpp b/compiler-rt/lib/nsan/nsan.cpp
index 7a5f013579dfb..efcbf6a1bedb4 100644
--- a/compiler-rt/lib/nsan/nsan.cpp
+++ b/compiler-rt/lib/nsan/nsan.cpp
@@ -535,7 +535,10 @@ int32_t checkFT(const FT value, ShadowFT Shadow, CheckTypeT CheckType,
   }
 
   if (flags().halt_on_error) {
-    Printf("Exiting\n");
+    if (common_flags()->abort_on_error)
+      Printf("ABORTING\n");
+    else
+      Printf("Exiting\n");
     Die();
   }
   return flags().resume_after_warning ? kResumeFromValue : kContinueWithShadow;
@@ -776,6 +779,16 @@ extern "C" SANITIZER_INTERFACE_ATTRIBUTE void __nsan_dump_shadow_args() {
   printf("args tag: %lx\n", __nsan_shadow_args_tag);
 }
 
+static void OnStackUnwind(const SignalContext &sig, const void *,
+                          BufferedStackTrace *stack) {
+  stack->Unwind(StackTrace::GetNextInstructionPc(sig.pc), sig.bp, sig.context,
+                common_flags()->fast_unwind_on_fatal);
+}
+
+static void NsanOnDeadlySignal(int signo, void *siginfo, void *context) {
+  HandleDeadlySignal(siginfo, context, GetTid(), &OnStackUnwind, nullptr);
+}
+
 bool __nsan::nsan_initialized;
 bool __nsan::nsan_init_is_running;
 
@@ -789,6 +802,9 @@ extern "C" SANITIZER_INTERFACE_ATTRIBUTE void __nsan_init() {
   InitializeSuppressions();
   InitializePlatformEarly();
 
+  DisableCoreDumperIfNecessary();
+  InstallDeadlySignalHandlers(NsanOnDeadlySignal);
+
   if (!MmapFixedNoReserve(TypesAddr(), UnusedAddr() - TypesAddr()))
     Die();
 

[MaskRay]

Disabling core dumps makes sense.

google/sanitizers#637 describes why many sanitizers install signal handlers. But nsan doesn't deal with these undefined behaviors and perhaps doesn't need it?

[MaskRay]

Disabling core dumps makes sense.

google/sanitizers#637 describes why many sanitizers install signal handlers. But nsan doesn't deal with these undefined behaviors and perhaps doesn't need it?

[chestnykh]

Disabling core dumps makes sense.

google/sanitizers#637 describes why many sanitizers install signal handlers. But nsan doesn't deal with these undefined behaviors and perhaps doesn't need it?

Okay, i thought that signal handler is installed to have detail report about crashes caused by signal :)

[alexander-shaposhnikov]

to me LG, i'd wait for @MaskRay

[chestnykh]

@MaskRay @alexander-shaposhnikov merge please this PR, I don't have write permissions