Nix Guide

[arunoruto]

More and more development is shifting to nix using the system (nixos) or the package manager (nix). It has perks compared to containerized development like docker. It even has a large package repository (over 80,000 packages as of writing), but it sometimes lacks the traction of large projects like oneAPI.

I was recently trying to get numba-dpex running on my Laptop (which runs Nixos), but in the end, I failed with the message ImportError: libintlc.so.5: cannot open shared object file: No such file or directory. I guess this happens to the missing oneAPI libs, which should be installed.

It would be nice to have a small guide on setting up a Nix system to run numba-dpex, or at least installing the required packages! There have been some previous attempts at packaging the one API into Nix in this issue, but sadly, it only works as a NUR (Nix User Repository, like AUR for Arch) package. Maybe someone here could help bring this to Nix and make it easier for us to utilize.

And sorry if this is the wrong repo to post this discussion. It is the one project I am most familiar with, and I do hope someone can help me a bit!

[diptorupd]

@arunoruto sorry I missed your question here. Thanks for your feedback. I do not think anyone has tried Nix to run numba-dpex before, so in a way you are a pioneer. 😄

Let me discuss with some of the other developers who are more experienced with Nix and get back to you.

[diptorupd]

@arunoruto I asked around and at the moment nothing officially is happening to support Nix. The dpcpp-cpp-rt compiler runtime and the intel compute runtime drivers will need to be packaged. I did see that some one has packaged the intel compute runtime, but nothing exists for the dpcpp-cpp-rt. Unfortunately, that is as far as I could get. I will talk to some others and try to get more information.

[ZzEeKkAa]

As @diptorupd said, there is no native support for nixos. While it may sound like overkill, but I bet it is possible to install conda inside nix and install dpex inside conda. There are instructions how to do so. More over intel provides oneapi inside wheels, but I'm not sure how easy it will be to get it work using something like pip.

[arunoruto]

I am currently using Poetry to manage virtual environments and packages, and I somehow succeeded in overcoming the error mentioned regarding libintlc.so.5: I had to put the virtual environment's lib dir into LD_LIBRARY_PATH, and it found it!

Nevertheless, it seems like neither dpctl, nor the oneAPI implementation in Julia can find my iGPU on NixOS. I guess I would need the OpenCL or LevelZero drivers installed for it to be found (or correct me if I am mistaken!)

[diptorupd]

Yes, that will be the first place to look at. Do you have this package installed? https://github.com/NixOS/nixpkgs/tree/master/pkgs/os-specific/linux/intel-compute-runtime.

The other possible problem might be access to the GPU for your user. On Ubuntu, to access Intel GPU devices your user id should be added to the render group. Do you know what is the corresponding way in NixOS?

[arunoruto]

Yes, that will be the first place to look at. Do you have this package installed? https://github.com/NixOS/nixpkgs/tree/master/pkgs/os-specific/linux/intel-compute-runtime.

I included it into my hardware.opengl.extraPackages list as mentioned in this thread.

The other possible problem might be access to the GPU for your user. On Ubuntu, to access Intel GPU devices your user id should be added to the render group. Do you know what is the corresponding way in NixOS?

I am not sure either how it is handled, but the render group is mentioned in /run/current-system/systemd/lib/udev/rules.d/50-udev-default.rules:

SUBSYSTEM=="drm", KERNEL!="renderD*", GROUP="video"
SUBSYSTEM=="drm", KERNEL=="renderD*", GROUP="render", MODE="0666"
SUBSYSTEM=="kfd", GROUP="render", MODE="0666"
SUBSYSTEM=="accel", GROUP="render", MODE="0666"

[diptorupd]

@arunoruto I right now have no understanding of Nix and how the packaging work. Specific to numba-dpex and dpctl, at this point the whole of oneAPI is not needed. At runtime both numba-dpex and dpctl require the dpcpp compiler runtime and OpenCL/LevelZero drivers. To build from source, the dpcpp compiler is needed.

One detour that I can suggest is to build a standalone Nix package for the dpcpp compiler. It is even possible to use the open-source Intel dpcpp compiler (https://github.com/intel/llvm/releases) instead of the proprietary version in oneAPI.

Once there is a dpcpp compiler package, dpctl and numba-dpex can be built using it. If you give it a shot and ask questions here, I and others can try and help you. I am also cross posting it to the NixOS issue you mentioned.

[arunoruto]

@diptorupd, thanks for the clarification and your guide!

Regarding the standalone dpcpp package, maybe someone with some experience could help me there.
@RossComputerGuy I hope you don't mind me tagging you here since I saw you contribute to the latest llvm 18 release in nixpkgs. Do you have any guidance on how to tackle this? I have never packaged something for Nix so far.

[oleksandr-pavlyk]

I think to triage why GPUs are not being seen by DPC++ runtime, one needs to run with SYCL_PI_TRACE=1, here is what I see on Ubuntu 20.04 in WSL:

(dev_dpctl) opavlyk@opavlyk-mobl:~/repos/dpctl$ SYCL_PI_TRACE=1 python -m dpctl -s
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_opencl.so [ PluginVersion: 14.39.1 ]
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_level_zero.so [ PluginVersion: 14.39.1 ]
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_unified_runtime.so [ PluginVersion: 14.39.1 ]
Intel(R) FPGA Emulation Platform for OpenCL(TM) OpenCL 1.2 Intel(R) FPGA SDK for OpenCL(TM), Version 20.3
Intel(R) OpenCL OpenCL 3.0 LINUX
Intel(R) OpenCL Graphics OpenCL 3.0
Intel(R) Level-Zero 1.3

In addition to DCP++ runtimes NixOS also needs to ensure that driver loaders are available. For Intel GPUs that would be level-zero loader (https://github.com/oneapi-src/level-zero).

Level-Zero loaders would attempt to load a user-mode linux library libze_intel_gpu.so. If loading is failed (library not found, DL library can not resolve symbols, etc.), or initialization function call returns an error, Level-Zero device would not be reported as available.

libze_intel_gpu itself works with kernel mode driver that must be installed and active, and have matching version for things to work correctly, but if you are able to see and use GPU devices outside of NixOS, than that part of it is in order. What I'm trying to stress here there is a tight coupling between user-mode libraries and kernel-mode modules that must be preserved.

One way to find out whether the failure to see GPU devices is due to library version mismatch (symbols not found during SO loading) is to run python -m dpctl -s with environment variable setting LD_DEBUG=libs. This is going to generate a large volume of output, so it is best to redirect it to a file. You need to be looking for loading of "libze_intel_gpu" library and whether its initialization was successful.

[arunoruto]

It seems like the level-zeros library is now available in nixpkgs. I will try to look into enabling DCP++ from the open-source llvm repository.

I was trying recently to make dpex run, but it always says that the libsycl.so library is missing.
DPC++ is needed for oneAPI, so it would have been needed either way, so why not make it a standalone nix package :)

[ZzEeKkAa]

@arunoruto it would be nice to have open source build of compiler, but it may be tricky to achieve. Refer to github CI's how the cmake build folder is generated using python script if you convinced you want to go this way. I would recommend starting from adapting offline installer: https://www.intel.com/content/www/us/en/developer/tools/oneapi/dpc-compiler-download.html?operatingsystem=linux&distribution-linux=offline

[ZzEeKkAa]

You can find libsycl.so at intel-sycl-rt, for example from pypi repo: https://pypi.org/project/intel-sycl-rt/ . However you may need to update CPATH to python's library folder.