-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add work product for GSoC'23: Expanding binary compatibility mode
Signed-off-by: Tianyi Liu <[email protected]>
- Loading branch information
Showing
1 changed file
with
137 additions
and
0 deletions.
There are no files selected for viewing
137 changes: 137 additions & 0 deletions
137
gsoc-2023/work_product/Liu-Extending-Binary-Compatibility/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| # Unikraft GSoC'23: Expanding binary compatibility mode | ||
|
|
||
| <img width="100px" src="https://summerofcode.withgoogle.com/assets/media/gsoc-2023-badge.svg" align="right" /> | ||
|
|
||
| ## Summary | ||
|
|
||
| One of the weak points of most unikernel projects has always been application support, often requiring that applications be ported to the target unikernel OS. | ||
| In this area, Unikraft has made significant progress by prioritizing POSIX compatibility. | ||
| Through its binary compatibility mode, Unikraft allows unmodified ELFs to be executed, with resulting syscalls wrapped and redirected to the Unikraft core via the [`app-elfloader`](https://github.com/unikraft/app-elfloader). | ||
| Currently, Unikraft supports loading both statically and dynamically linked ELFs in binary compatibility mode. | ||
| However, Unikraft has not been able to fully support all behaviors of glibc and user applications, which causes certain programs to fail in initializing or crash. | ||
| Additionally, running applications in binary compatibility mode incurs performance loss compared to ported programs, due to the TLS switching and system interrupts caused by binary syscalls. | ||
| To address the issues mentioned above, this project focuses on expanding Unikraft's binary compatibility mode by: | ||
|
|
||
| * Use [`vDSO`](https://man7.org/linux/man-pages/man7/vdso.7.html) (_virtual dynamic shared object_) and `vsyscall` (_virtual system call_) to bypass binary syscalls. | ||
|
|
||
| * Expanding supported syscalls to provide compatibility for a wider range of applications. | ||
|
|
||
| * Adding AArch64 architecture support for `app-elfloader` and `syscall-shim` layer. | ||
|
|
||
| * Establishing a CI/CD system to ensure that new commits do not break libc compatibility. | ||
|
|
||
| ## GSoC contributor | ||
|
|
||
| Name: `Tianyi Liu` | ||
|
|
||
| Email: `[email protected]` | ||
|
|
||
| Github profile: [i-Pear](https://github.com/i-Pear/) | ||
|
|
||
| ## Mentors | ||
|
|
||
| [Razvan Deaconescu](https://github.com/razvand) | ||
|
|
||
| [Simon Kuenzer](https://github.com/skuenzer) | ||
|
|
||
| [Stefan Jumarea](https://github.com/StefanJum) | ||
|
|
||
| ## vDSO and vsyscall Support for Binary Compatibility | ||
|
|
||
| The `vsyscall` (_virtual system call_) is the first and oldest mechanism used to accelerate system calls. | ||
| Due to security concerns, it has been deprecated, and `vDSO` (_virtual dynamic shared object_) serves as its successor. | ||
| However, in the context of unikernels, kernel isolation is not a concern, allowing us to utilize both of them. | ||
| The [`app-elfloader`](https://github.com/unikraft/app-elfloader) now supports both of these features. | ||
|
|
||
| If you don't have the source code of your application but desire the performance of Unikraft, [`app-elfloader`](https://github.com/unikraft/app-elfloader) can load unmodified ELF files. | ||
| However, there will be some performance overhead compared to a native build, while the two newly introduced features can help mitigate this difference. | ||
|
|
||
| The `vDSO` is available for both dynamically linked applications and statically linked ones, to accelerate certain commonly used time-related system calls (`clock_gettime`, `gettimeofday`, `time`, `clock_getres`). | ||
| As a result, some server programs can experience significant performance improvements. | ||
|
|
||
| If your application is dynamically linked, it can further utilize `vsyscall` to avoid costly binary syscalls (since `vsyscall` has been deprecated in the latest version of libc, we provide patched dynamic runtime libraries for [`glibc`](https://github.com/unikraft/fork-glibc) and [`musl`](https://github.com/unikraft/fork-musl)). | ||
|
|
||
| ## Adapting to the .NET Runtime | ||
|
|
||
| `.Net` is a widely used framework. | ||
| According to TIOBE's rankings, `C#` and `Visual Basic`, both based on `.NET`, have market shares of `6.87%` and `2.90%` respectively. | ||
| Therefore, adapting to the `.Net` runtime can significantly increase the number of applications supported by Unikraft. | ||
|
|
||
| As a complex dynamic language virtual machine, the `.NET` runtime requires a lot of configuration in the environment, involving many system calls that require support from the operating system. | ||
| However, one advantage is that the `.NET` runtime is implemented in C++ and directly uses `glibc`, so our previous support for `glibc` can be reused, including `VDSO`. | ||
|
|
||
| Specifically, the `.NET` runtime relies on the following support: | ||
|
|
||
| For the `posix-mmap` module, it requires the `mlock` and `msync` system calls. | ||
| However, since Unikraft doesn't support swap and doesn't write back the content of mmap to files, both of these system calls can be safely ignored. | ||
|
|
||
| For the `vfscore` module, the `mknodat` system call is needed to create the debug pipe `/tmp/clr-debug-pipe`. | ||
| We found that adding the environment variable `COMPlus_EnableDiagnostics=0` can prevent this behavior. | ||
| Hence, we decided not to implement the `mknodat` system call to avoid disrupting compatibility with other applications. | ||
| We also compiled a modified `.NET` runtime version with the default value of `COMPlus_EnableDiagnostics` changed to 0. | ||
| Both the original and modified binary libraries are stored in the `dynamic-apps` repository. | ||
|
|
||
| For the `uksched` module, it requires the `sched_getaffinity` and `sched_setaffinity` system calls. | ||
| Since Unikraft's `SMP`support is still under developing, we temporarily ignore `sched_setaffinity` and ensure that `sched_getaffinity` always returns `CPU0`. | ||
|
|
||
| Finally, The `.Net` runtime uses [`__pthread_getattr_np`](https://github.com/bminor/glibc/blob/glibc-2.37/nptl/pthread_getattr_np.c#L85) | ||
| in [`CPalThread::GetStackBase()`](https://github.com/dotnet/runtime/blob/v7.0.7/src/coreclr/pal/src/thread/thread.cpp#L2683) | ||
| and [`CPalThread::GetStackLimit()`](https://github.com/dotnet/runtime/blob/v7.0.7/src/coreclr/pal/src/thread/thread.cpp#L2723). | ||
| In [`__pthread_getattr_np`](https://github.com/bminor/glibc/blob/glibc-2.37/nptl/pthread_getattr_np.c#L85), | ||
| the `glibc` implementation reads `/proc/self/maps` to [get stack addresses](https://github.com/bminor/glibc/blob/glibc-2.37/nptl/pthread_getattr_np.c#L129). | ||
| Instead of relying on identifying the names of mappings, it checks if each memory mapping includes the known address range of the stack. | ||
| Since `procfs` is not yet implemented, users have to change the content of `/proc/1/maps` in their `rootfs`, to make the address range of `[stack]` contains the real stack area. | ||
| Otherwise, the runtime will crash with `Failed to create CoreCLR, HRESULT: 0x8007000E`. | ||
| To get to know what's the stack address, the debug information provided in [app-elfloader](https://github.com/unikraft/app-elfloader/blob/bbb92f8c04bd58f0cf52b9e76250b0e03c5fc7e7/main.c#L239) may be helpful. | ||
|
|
||
| ## AArch64 architecture support for `app-elfloader` | ||
|
|
||
| To support binary syscalls for the AArch64 architecture, I introduced an adapter function into the AArch64 interrupt handling module, redirecting system calls to the `syscall-shim` layer. | ||
| After testing, this has proven to work effectively. | ||
| However, it's worth noting that there are many libraries within Unikraft that do not offer complete support for the AArch64 architecture, which will require further effort. | ||
|
|
||
| ## PRs opened by me | ||
|
|
||
| * [[unikraft] AArch64 binary syscall support](https://github.com/unikraft/unikraft/pull/1009) | ||
|
|
||
| * [[app-elfloader] AArch64 support](https://github.com/unikraft/app-elfloader/pull/24) | ||
|
|
||
| * [[Static-pie-apps] Node.js static build](https://github.com/unikraft/static-pie-apps/pull/85) | ||
|
|
||
| * [[dynamic-apps] Busybox dynamic build](https://github.com/unikraft/dynamic-apps/pull/17) | ||
|
|
||
| * [[dynamic-apps] .Net 7.0.7 dynamic build](https://github.com/unikraft/dynamic-apps/pull/60) | ||
|
|
||
| * [[unikraft] Add dependencies for dotnet runtime](https://github.com/unikraft/unikraft/pull/1004) | ||
|
|
||
| * [[app-elfloader] VDSO & vsyscall support](https://github.com/unikraft/app-elfloader/pull/23) | ||
|
|
||
| * [[fork-musl] vsyscall support for musl](https://github.com/unikraft/fork-musl/pull/1) | ||
|
|
||
| * [[fork-glibc] vsyscall support for glibc](https://github.com/unikraft/fork-glibc/pull/1) | ||
|
|
||
| * [[unikraft] A tiny fix for build system](https://github.com/unikraft/unikraft/pull/817) | ||
|
|
||
| ## Journals | ||
|
|
||
| The success of this project wouldn't have been possible without the help of community members. | ||
| We've engaged in numerous relevant discussions. | ||
| For details you can check our meeting notes in the [Unikraft meeting notes repo](https://github.com/unikraft/meeting-notes/tree/staging/discussions/app-compat) and on [discord channel](https://discord.com/channels/762976922531528725/1118149304717160508). | ||
|
|
||
| ## Blog posts | ||
|
|
||
| During GSoC'23, I provided 3 blog posts which offer a comprehensive status for the 3 months that I was involved with Unikraft: | ||
|
|
||
| * [1st blog post: Project proposal and initial steps](https://unikraft.org/blog/2023-06-23-unikraft-gsoc-app-compat-1/) | ||
|
|
||
| * [2nd blog post: Design details and history of vDSO](https://github.com/unikraft/docs/pull/292/files) | ||
|
|
||
| * [3rd blog post: Adaptation process for .NET 7.0](https://github.com/unikraft/docs/pull/300/files) | ||
|
|
||
| ## Acknowledgement | ||
|
|
||
| I would like to express my sincere gratitude to my mentors and the Unikraft community for their support throughout my participation in the project. | ||
| Throughout the course of this project, I've significantly deepened my understanding of unikernels. | ||
| Also, Unikraft's modular architecture and streamlined implementation have provided me with a clearer insight into how kernels function, enhancing my proficiency in kernel development. | ||
| Throughout the project, the community has always been a tremendous source of support and valuable insights. | ||
| I feel fortunate to have had the opportunity to work with all of you. |