runsc unable to run bash from a guix pack

[chetan-reddy]

Description

I'm trying to use runsc to run a software bundle built by guix pack. While it works if i use /bin/sleep 100 as the command, runsc fails immediately if i use /bin/bash -c "sleep 100" as the command. I first encountered this behavior using podman(where I verified that bash from the pack works with runc and only fails with runsc), but I've since reproduced this using just runsc
I've supplied the steps to reproduce and attached the logs below. I'm running guix on a Debian 12 box. Please feel free to close if this is not a supported config or if it's not worth investigating. The first error in the log is this line:

Unhandled user fault: addr=7ee9f3a6e025 ip=7ee9f3a6e025 access=r-x sig=11 err=bad address

Steps to reproduce

$ guix describe
Generation 4    Aug 13 2024 11:24:22    (current)
  guix 99a81b6
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 99a81b6f2a1386d6ea76a6ecb1942f1f7f891080
$ guix pack -S /bin=bin coreutils bash
/gnu/store/j50l1pz73w5sdncw43b6j1xzsd9l2bpx-coreutils-bash-tarball-pack.tar.gz
$ mkdir bundle
$ cd bundle/
$ mkdir --mode=0755 rootfs
$ sudo tar -xzf /gnu/store/j50l1pz73w5sdncw43b6j1xzsd9l2bpx-coreutils-bash-tarball-pack.tar.gz -C rootfs --same-owner --same-permissions
$ ls rootfs/
bin  gnu
$ sudo runsc.20240820 spec -- /bin/bash -c "sleep 100"
$ ls
config.json  rootfs
$ sudo runsc.20240820 -debug -debug-log=/tmp/runsc.debug.log run test
$ echo $?
139

runsc version

$ runsc.20240820 -version
runsc version release-20240820.0
spec: 1.1.0-rc.1

docker version (if using docker)

No response

uname

Linux mini2011 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux

kubectl (if using Kubernetes)

No response

repo state (if built from source)

No response

runsc debug logs (if available)

No response

[chetan-reddy]

runsc.debug.log

[kevinGC]

Would you mind trying this with the --platform=ptrace flag? Just to help determine whether the platform is part of the issue.

[chetan-reddy]

Would you mind trying this with the --platform=ptrace flag? Just to help determine whether the platform is part of the issue.

It does work as expected with platform=ptrace . Let me know if you need debug logs for the working platform=ptrace run. Could it be related to the fact that I'm running gvisor on really old hardware (a macmini from 2011)? I will check on different hardware (from 2014) later today and get back.

[chetan-reddy]

I just tried on a 2014 macbook air 6,2 (again running Debian 12), and both platform=ptrace and platform=kvm work as expected, but platform=systrap fails with the same bad address error.

I've uploaded the Guix pack to my google drive. Here's the link https://drive.google.com/file/d/1NL6fu_h6HmUQAkvhZrzNxjFgKR9v5lXX/view?usp=sharing

[avagin]

could you show /proc/cpuinfo from this machine?

[chetan-reddy]

macbookair2014_cpuinfo.txt
macmini2011_cpuinfo.txt

I have attached /proc/cpuinfo from both the machines on which it fails with platform=systrap ; the debug log from yesterday corresponds to macmini2011_cpuinfo.txt

[chetan-reddy]

Not sure if it's relevant, but platform=kvm never worked on the 2011 macmini. runsc-sandbox just spins at 100% of cpu. I figured it's because the hardware is old so I didn't bother to report, especially since platform=systrap has been working well for over a year (until I tried the guix pack yesterday). On the 2014 macbookair, platform=kvm has been working well. Unfortunately I don't have any newer hardware running linux to check right now, but if this issue is because my hardware is old, then please feel free to close. I'll stick to the platforms that do work for my experiments with guix packs.

[avagin]

@chetan-reddy I tried to run your repro steps in a gce vm and everything works as expected.

Could run you the rerproducer with the enabled strace (--strace) and attach logs for both ptrace and systrap platfroms?

[chetan-reddy]

runsc.ptrace.log
runsc.systrap.log

@avagin attached the logs with strace from the 2011 mac mini. Let me know if you also want the logs from the 2014 macbook air.

[avagin]

@chetan-reddy could you try to recompile runsc with this patch:

diff --git a/pkg/sentry/platform/systrap/subprocess.go b/pkg/sentry/platform/systrap/subprocess.go
index 11f08edba..3febf7780 100644
--- a/pkg/sentry/platform/systrap/subprocess.go
+++ b/pkg/sentry/platform/systrap/subprocess.go
@@ -837,7 +837,7 @@ func (s *subprocess) switchToApp(c *platformContext, ac *arch.Context64) (isSysc
        ctxState := ctx.state()
        if ctxState == sysmsg.ContextStateSyscallCanBePatched {
                ctxState = sysmsg.ContextStateSyscall
-               shouldPatchSyscall = true
+               shouldPatchSyscall = false
        }
 
        if ctxState == sysmsg.ContextStateSyscall || ctxState == sysmsg.ContextStateSyscallTrap {

[avagin]

@chetan-reddy I reproduced the issue in my environment. The syscall patching doesn't handle the next code properly:

00000000000fb5e0 <ioctl>:
__GI___ioctl():
   fb5e0:       48 8d 44 24 08          lea    0x8(%rsp),%rax
   fb5e5:       48 89 54 24 e0          mov    %rdx,-0x20(%rsp)
   fb5ea:       48 89 44 24 c0          mov    %rax,-0x40(%rsp)
   fb5ef:       48 8d 44 24 d0          lea    -0x30(%rsp),%rax
   fb5f4:       48 89 44 24 c8          mov    %rax,-0x38(%rsp)
   fb5f9:       b8 10 00 00 00          mov    $0x10,%eax
   fb5fe:       c7 44 24 b8 10 00 00    movl   $0x10,-0x48(%rsp)
   fb605:       00 
   fb606:       0f 05                   syscall

It revises five bytes before the syscall instruction and if they match the proper mov $sysno, %rax instruction, it replace both instructions with the jmp instruction. In this case, it doesn't work because five bytes before syscall b8 10 00 00 00 looks like mov $0x10, %eax but actually it is another instruction. Finita la commedia...

[chetan-reddy]

Sweet @avagin . You just saved me a bunch of effort trying to build runsc in a guix shell container (using the origin/go branch since no access to docker). Can't wait for the day when Guix includes runsc.

[chetan-reddy]

@avagin I too can confirm that with your patch shouldPatchSyscall = false it works. Just in case it helps anyone in the future, the command to build runsc from the origin/go branch in Guix is:

~/dev/gvisor$ guix shell -F -N -C go nss-certs -- go build -o bin/ ./runsc