Crash during compile #3894

dkegel-fastly · 2023-09-07T18:04:06Z

This is on linux with tinygo 0.29 compiling a fairly small "hello, http" app.

Seems to be intermittent, but often enough to cause CI traffic jams.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x290 pc=0x5148bb2]

runtime stack:
runtime.throw({0x556e85f?, 0x40?})
        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f148c85f880 sp=0x7f148c85f850 pc=0x4f3f1d
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:825 +0x3e9 fp=0x7f148c85f8e0 sp=0x7f148c85f880 pc=0x50b1c9

goroutine 182 [syscall]:
runtime.cgocall(0x8517d0, 0xc002979fd0)
        /usr/local/go/src/runtime/cgocall.go:157 +0x5c fp=0xc002979fa8 sp=0xc002979f70 pc=0x4c36bc
tinygo.org/x/go-llvm._Cfunc_LLVMConstInt(0x7f148b916268, 0x0, 0x0)
        _cgo_gotypes.go:3065 +0x4d fp=0xc002979fd0 sp=0xc002979fa8 pc=0x6ef08d
tinygo.org/x/go-llvm.ConstInt.func1({0x853b80?}, 0xc00297a038?, 0x0)
        /go/pkg/mod/tinygo.org/x/[email protected]/ir.go:843 +0x5e fp=0xc00297a010 sp=0xc002979fd0 pc=0x70719e
tinygo.org/x/go-llvm.ConstInt({0x7f148b916268?}, 0xc004766240?, 0x60?)
        /go/pkg/mod/tinygo.org/x/[email protected]/ir.go:843 +0x19 fp=0xc00297a038 sp=0xc00297a010 pc=0x7070f9
github.com/tinygo-org/tinygo/compiler.(*compilerContext).getTypeCode(0xc002840280, {0x55da858, 0xc00507e560?})
        /__w/tinygo/tinygo/compiler/interface.go:444 +0x2f4e fp=0xc00297a568 sp=0xc00297a038 pc=0x7ab4ae
github.com/tinygo-org/tinygo/compiler.(*compilerContext).getTypeCode(0xc002840280, {0x55da830, 0xc0000da7e0?})
        /__w/tinygo/tinygo/compiler/interface.go:281 +0x257c fp=0xc00297aa98 sp=0xc00297a568 pc=0x7aaadc

This may be related to #3893

The text was updated successfully, but these errors were encountered:

dkegel-fastly · 2023-09-07T19:24:21Z

Another instance:

+ time tinygo build -opt 2 -target=wasi -gc=leaking -o bin/main.wasm
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100000019 pc=0x521bbd8]

runtime stack:
runtime.throw({0x556e85f?, 0x7efc0?})
        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7efcd1c21610 sp=0x7efcd1c215e0 pc=0x4f3f1d
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:825 +0x3e9 fp=0x7efcd1c21670 sp=0x7efcd1c21610 pc=0x50b1c9

goroutine 310 [syscall]:
runtime.cgocall(0x84eb50, 0xc00f021b30)
        /usr/local/go/src/runtime/cgocall.go:157 +0x5c fp=0xc00f021b08 sp=0xc00f021ad0 pc=0x4c36bc
tinygo.org/x/go-llvm._Cfunc_LLVMVerifyModule(0x7efcd068c2d0, 0x1, 0xc008cd21c0)
        _cgo_gotypes.go:9317 +0x4c fp=0xc00f021b30 sp=0xc00f021b08 pc=0x6fb72c
tinygo.org/x/go-llvm.VerifyModule.func1({0x70373f?}, 0xd068c2d0?, 0x7efcd0f64010?)
        /go/pkg/mod/tinygo.org/x/[email protected]/analysis.go:38 +0x7d fp=0xc00f021b78 sp=0xc00f021b30 pc=0x6fbc5d
tinygo.org/x/go-llvm.VerifyModule({0xc0003e4910?}, 0x53?)
        /go/pkg/mod/tinygo.org/x/[email protected]/analysis.go:38 +0x45 fp=0xc00f021bd0 sp=0xc00f021b78 pc=0x6fbb25
github.com/tinygo-org/tinygo/builder.Build.func3(0xc00d93f920)
        /__w/tinygo/tinygo/builder/build.go:363 +0x397 fp=0xc00f021f78 sp=0xc00f021bd0 pc=0x8005b7
github.com/tinygo-org/tinygo/builder.runJob(0xc00d93f920, 0x0?)
        /__w/tinygo/tinygo/builder/jobs.go:222 +0x4f fp=0xc00f021fc0 sp=0xc00f021f78 pc=0x80c84f
github.com/tinygo-org/tinygo/builder.runJobs.func2()
        /__w/tinygo/tinygo/builder/jobs.go:123 +0x2a fp=0xc00f021fe0 sp=0xc00f021fc0 pc=0x80c18a
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00f021fe8 sp=0xc00f021fe0 pc=0x5266c1
created by github.com/tinygo-org/tinygo/builder.runJobs
        /__w/tinygo/tinygo/builder/jobs.go:123 +0x5be

aykevl · 2023-09-10T17:09:04Z

Can you try reproducing this crash with:

the race condition checker enabled (-race)
LLVM built with assertions enabled (remove llvm-build and run make llvm-build ASSERT=1, then build TinyGo using make, possibly modifying the Makefile to add -race)

That should result in a much more useful crash report.

dkegel-fastly · 2023-09-12T17:34:43Z

possibly interesting warning during compile:

[2023-09-12T17:05:51.808Z] In constructor 'llvm::MachineModuleInfo::MachineModuleInfo(const llvm::LLVMTargetMachine*)',
[2023-09-12T17:05:51.808Z]     inlined from 'llvm::MachineModuleInfoWrapperPass::MachineModuleInfoWrapperPass(const llvm::LLVMTargetMachine*)' at /tinygo.tmp/tinygo/llvm-project/llvm/lib/CodeGen/MachineModuleInfo.cpp:181:26,
[2023-09-12T17:05:51.808Z]     inlined from 'llvm::Pass* llvm::callDefaultCtor() [with PassName = llvm::MachineModuleInfoWrapperPass]' at /tinygo.tmp/tinygo/llvm-project/llvm/include/llvm/PassSupport.h:80:76:
[2023-09-12T17:05:51.808Z] /tinygo.tmp/tinygo/llvm-project/llvm/lib/CodeGen/MachineModuleInfo.cpp:76:51: warning: 'this' pointer is null [-Wnonnull]
[2023-09-12T17:05:51.808Z]    76 |   Context.setObjectFileInfo(TM->getObjFileLowering());
[2023-09-12T17:05:51.808Z]       |                             ~~~~~~~~~~~~~~~~~~~~~~^~

dkegel-fastly · 2023-09-12T19:21:19Z

OK, here's a log of a clean, fully from source build with assertions enabled via

make llvm-build ASSERT=1

and with

sed -i 's/build -buildmode/build -race -buildmode/' Makefile

to enable race detection:

[2023-09-12T19:09:55.656Z] SIGSEGV: segmentation violation
[2023-09-12T19:09:55.656Z] PC=0x5af1577 m=5 sigcode=1
[2023-09-12T19:09:55.656Z] signal arrived during cgo execution
[2023-09-12T19:09:55.656Z]
[2023-09-12T19:09:55.656Z] goroutine 343 [syscall]:
[2023-09-12T19:09:55.656Z] runtime.cgocall(0xa983b0, 0xc027af3a50)
[2023-09-12T19:09:55.656Z]      /usr/local/go/src/runtime/cgocall.go:157 +0x5b fp=0xc027af3a28 sp=0xc027af39f0 pc=0x4b0f9b
[2023-09-12T19:09:55.656Z] tinygo.org/x/go-llvm._Cfunc_LLVMVerifyModule(0x7f6514025520, 0x1, 0xc01ba20810)
[2023-09-12T19:09:55.656Z]      _cgo_gotypes.go:9323 +0x7b fp=0xc027af3a50 sp=0xc027af3a28 pc=0x8478bb
[2023-09-12T19:09:55.656Z] tinygo.org/x/go-llvm.VerifyModule.func1({0x51a289?}, 0x27af3ae0?, 0xa2f465?)
[2023-09-12T19:09:55.656Z]      /root/go/pkg/mod/tinygo.org/x/[email protected]/analysis.go:38 +0x86 fp=0xc027af3a98 sp=0xc027af3a50 pc=0x847fe6
[2023-09-12T19:09:55.656Z] tinygo.org/x/go-llvm.VerifyModule({0xc000028780?}, 0x46?)
[2023-09-12T19:09:55.656Z]      /root/go/pkg/mod/tinygo.org/x/[email protected]/analysis.go:38 +0x70 fp=0xc027af3af0 sp=0xc027af3a98 pc=0x847e70
[2023-09-12T19:09:55.656Z] github.com/tinygo-org/tinygo/builder.Build.func3(0xc0126d92c0)
[2023-09-12T19:09:55.656Z]      /tinygo.tmp/tinygo/builder/build.go:371 +0x4f5 fp=0xc027af3f40 sp=0xc027af3af0 pc=0xa2ddf5
[2023-09-12T19:09:55.656Z] github.com/tinygo-org/tinygo/builder.runJob(0xc0126d92c0, 0x0?)
[2023-09-12T19:09:55.656Z]      /tinygo.tmp/tinygo/builder/jobs.go:222 +0x76 fp=0xc027af3fb0 sp=0xc027af3f40 pc=0xa40e16
[2023-09-12T19:09:55.656Z] github.com/tinygo-org/tinygo/builder.runJobs.func2()
[2023-09-12T19:09:55.656Z]      /tinygo.tmp/tinygo/builder/jobs.go:123 +0x45 fp=0xc027af3fe0 sp=0xc027af3fb0 pc=0xa40485
[2023-09-12T19:09:55.656Z] runtime.goexit()
[2023-09-12T19:09:55.656Z]      /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc027af3fe8 sp=0xc027af3fe0 pc=0x517401
[2023-09-12T19:09:55.656Z] created by github.com/tinygo-org/tinygo/builder.runJobs in goroutine 10
[2023-09-12T19:09:55.656Z]      /tinygo.tmp/tinygo/builder/jobs.go:123 +0x836

dkegel-fastly · 2023-09-12T19:22:55Z

cf. #3874

kenbell · 2023-09-12T19:51:49Z

If it helps... I just randomly got this one:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
SIGABRT: abort
PC=0x7f05fbe90ffb m=19 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 215 [syscall]:
runtime.cgocall(0x8b1590, 0xc009127b30)
        /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc009127b08 sp=0xc009127ad0 pc=0x4dedeb
tinygo.org/x/go-llvm._Cfunc_LLVMVerifyModule(0x7f0588064310, 0x1, 0xc009d60468)
        _cgo_gotypes.go:9323 +0x4b fp=0xc009127b30 sp=0xc009127b08 pc=0x72d10b
tinygo.org/x/go-llvm.VerifyModule.func1({0xc009127b98?}, 0x9127b98?, 0xc009127bc0?)
        /home/kenneth/go/pkg/mod/tinygo.org/x/[email protected]/analysis.go:38 +0x77 fp=0xc009127b78 sp=0xc009127b30 pc=0x72d617
tinygo.org/x/go-llvm.VerifyModule({0xc0003442b8?}, 0x5b?)
        /home/kenneth/go/pkg/mod/tinygo.org/x/[email protected]/analysis.go:38 +0x3d fp=0xc009127bd0 sp=0xc009127b78 pc=0x72d4dd
github.com/tinygo-org/tinygo/builder.Build.func3(0xc00543d4a0)
        /home/kenneth/work/tinygo/interp-max-depth/builder/build.go:371 +0x370 fp=0xc009127f78 sp=0xc009127bd0 pc=0x861df0
github.com/tinygo-org/tinygo/builder.runJob(0xc00543d4a0, 0xc00006a798?)
        /home/kenneth/work/tinygo/interp-max-depth/builder/jobs.go:222 +0x4d fp=0xc009127fc0 sp=0xc009127f78 pc=0x86f5cd
github.com/tinygo-org/tinygo/builder.runJobs.func2()
        /home/kenneth/work/tinygo/interp-max-depth/builder/jobs.go:123 +0x25 fp=0xc009127fe0 sp=0xc009127fc0 pc=0x86ef25
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc009127fe8 sp=0xc009127fe0 pc=0x542541
created by github.com/tinygo-org/tinygo/builder.runJobs in goroutine 20
        /home/kenneth/work/tinygo/interp-max-depth/builder/jobs.go:123 +0x5bd

dkegel-fastly · 2023-09-12T20:10:06Z

wow, same stack exactly.

I should mention I got the error on the first try, and it's easy for me to run, so happy to test any fixes. (I'm already testing with the fix for #3906.)

dkegel-fastly · 2023-09-12T21:48:20Z

let me know if you need me to git bisect tinygo to see when it started happening

This usually works by chance, but leads to crashes. So we should never ever do this. I'm pretty sure this is the crash behind this issue: #3894 It may also have caused this crash: #3874 I have a suspicion this is also behind the rather crash-prone CircleCI jobs, that we haven't been able to find the source of. But we'll find out soon enough once this fix is merged. To avoid hitting this issue again in the future, I've created a PR to remove these dangerous functions altogether from the go-llvm API: tinygo-org/go-llvm#54

aykevl · 2023-09-18T21:35:19Z

I'm pretty sure the first crash is fixed by #3917.
Other crashes may also be fixed, it's hard to know for sure.

This usually works by chance, but leads to crashes. So we should never ever do this. I'm pretty sure this is the crash behind this issue: #3894 It may also have caused this crash: #3874 I have a suspicion this is also behind the rather crash-prone CircleCI jobs, that we haven't been able to find the source of. But we'll find out soon enough once this fix is merged. To avoid hitting this issue again in the future, I've created a PR to remove these dangerous functions altogether from the go-llvm API: tinygo-org/go-llvm#54

deadprogram · 2023-09-21T08:43:40Z

This is part of v0.30.0 release so now closing. Thanks!

aykevl mentioned this issue Sep 18, 2023

compiler: don't use types in the global context #3917

Merged

deadprogram added bug Something isn't working next-release Will be part of next release core labels Sep 19, 2023

deadprogram closed this as completed Sep 21, 2023

deadprogram removed the next-release Will be part of next release label Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash during compile #3894

Crash during compile #3894

dkegel-fastly commented Sep 7, 2023 •

edited

Loading

dkegel-fastly commented Sep 7, 2023

aykevl commented Sep 10, 2023

dkegel-fastly commented Sep 12, 2023

dkegel-fastly commented Sep 12, 2023 •

edited

Loading

dkegel-fastly commented Sep 12, 2023

kenbell commented Sep 12, 2023

dkegel-fastly commented Sep 12, 2023

dkegel-fastly commented Sep 12, 2023

aykevl commented Sep 18, 2023

deadprogram commented Sep 21, 2023

Crash during compile #3894

Crash during compile #3894

Comments

dkegel-fastly commented Sep 7, 2023 • edited Loading

dkegel-fastly commented Sep 7, 2023

aykevl commented Sep 10, 2023

dkegel-fastly commented Sep 12, 2023

dkegel-fastly commented Sep 12, 2023 • edited Loading

dkegel-fastly commented Sep 12, 2023

kenbell commented Sep 12, 2023

dkegel-fastly commented Sep 12, 2023

dkegel-fastly commented Sep 12, 2023

aykevl commented Sep 18, 2023

deadprogram commented Sep 21, 2023

dkegel-fastly commented Sep 7, 2023 •

edited

Loading

dkegel-fastly commented Sep 12, 2023 •

edited

Loading