Using full GL with android drivers. [Currently pure theory]

[twaik]

Hi. It is purely theoretical material, you can ignore it if you want or add something I miss.
I am not enough skilled to implement this but I found some things that can be useful for someone who is enough skilled.

Using full GL with Android's GLES library.

Everybody knows that we can emulate GL with virglrenderer+GLES and mesa's libGL.so+virpipe. But this virtio solution gives significant overhead. Related issue.
I think it is possible to get rid of virtio and dispatch GL (or Gallium, I am not so sure) directly to GLES. zzyiwei said it is possible to do something like GL->virgl->vrend->GLES to achieve this.

Using Android's GLES with X11 without exposing private API.

Forcing mesa to output to shared texture.

Currently Android's EGL library does not support X11, and there is no GLX support. We can use mesa but it can not use Android's drivers. There are only swrast and virpipe backends available. Both of them use shared memory fragment created with shmat/shmctl/shmdt/etc so mesa renders image to some internal buffer, memcpy's it to shared memory fragment and than displays it on X server (swrast) or memcpy's image from memfd/ashmem fragment to the same shared fragment and than displays it on X server (virpipe). This solution is universal but not much effective. Android 8+ lets us use AHardwareBuffer. We can allocate it an bind to GLES texture. We can not build ANativeWindow from such buffer without exposing private Android APIs, but it is possible to modify mesa/vrend source to force output to this texture instead of rendering to EGLSurface.

Binding shared texture (AHardwareBuffer) to X server.

As I said before Android's EGL is not compatible with X11 windows. But we can bind Android's AHardwareBuffer to new pixmap of X server and display it (and of cource we should force GL library to output everything to this AHardwareBuffer based texture). That is how I see this.

1. We can not get DMABUF using APIs exposed by NDK directly. But we can pass AHardwareBuffer through Unix socket with AHardwareBuffer_sendHandleToUnixSocket. So you can create socket with socketpair and pass AHardwareBuffer there. After that you will be able to recvmsg file descriptors related to this AHardwareBuffer from second file descriptor of this socket pair. As far as I know first file descriptor of AHardwareBuffer we get this way is always DMABUF file descriptor. This file descriptor can be mmaped as any ordinary file descriptor.
2. The second step is binding DMABUF to X11 pixmap. Xlib will not let us do this, but XCB does. There is xcb_shm_attach_fd function (man) that lets us bind any file descriptor to X server as memory fragment (just like XShmCreateImage, but not yet binding to XImage). And there is second function xcb_shm_create_pixmap (man) to create pixmap from this memory fragment. As you can see these functions are not properly documented, but they work.
3. Third step should be done is displaying the pixmap. NotKit's libhybris-based solution displays pixmap with xcb_present_pixmap on X11 window at coordinate (0,0) (here). That is pretty elegant and enough optimized I think.
4. Also Notkit's libhybris-based solution can subscribe to PRESENT extension's configure (window resize) events (here) and read them without interfering with the rest of application code (here).

Presenting pixmap should be done at every glXSwapBuffers call, and we should have at least two or three buffers to avoid flickering.

Using full GL with Android's Vulkan library with Zink.

Modern Zink requires vkCreateXcbSurfaceKHR for its Kopper WSI. Android's Vulkan library does not provide it. But I believe we can make Zink output to some offscreen surface with vkCreateDisplayPlaneSurfaceKHR and then output to shared memory fragment just like swrast do, or output to AHardwareBuffer bound to Vulkan as a texture and bind it to X11 pixmap (in the way I described for GLES output).
UPD: As far as I understand this we can not inject vkImages created with AHardwareBuffers to Vulkan swapchain, but we can replace the whone zink_kopper.c code with something more termux-specific, implementing output to a few AHardwareBuffers, with presenting these buffers to X11 window with Present X11 extension...

Sorry for my bad English.

[twaik]

@licy183 can you please ping someone who can help us?

[twaik]

I am likely doing something wrong, but my code is pretty useless. I am always getting Segmentation fault somewhere in /vendor/lib64/libIMGegl.so. Maybe that is because I am using Android 13, maybe it happens because I have errors in code. But it does not work. Implementing libEGL without exposing Android APIs must be more stable than this.
egl.tar.gz

[twaik]

Can we use eglCreatePbufferFromClientBuffer multiple times to emulate front, middle and backbuffer and swap them in wrapped eglSwapBuffers? This way AHardwareBuffer will be used as render target and all other functions should work as expected, without multiple changes in both libEGL and libGLESv2.so like in case of emulating wsi with glBindRenderbuffer and glBindFramebuffer.

[tareksander]

The ANDROID_get_native_client_buffer extension only specified using the ClientBuffer for an EGLImage, not eglCreatePbufferFromClientBuffer, so using it like that would likely be undefined behaviour. We could try mailing the author of the extension at google for clarification of whether that usage is supported.

[tareksander]

Nevermind, there isn't even a buffer type for createPbufferFromClientBuffer defined, so it won't work. I did find information about the GLES extension though, seems widely supported: https://opengles.gpuinfo.org/listreports.php?extension=GL_OES_EGL_image
Also judging from the fact that EGL_IMAGE_KHR and the extension to create an image from a HardwareBuffer are mandatory extensions, that should be supported widely. GL_OES_EGL_image_external is also mandatory, but we need framebuffer attachable textures, so it's not enough, but implementing full EGL image support isn't that big of a stretch for GLES vendors then.

[tareksander]

I looked a bit into X11, I can probably make my EGL wrapper render into a Pbuffer and then read that out into an X11 Pixmap. Anything more than that (that doesn't use private Android APIs) would require a custom X11 extension to exchange HardwareBuffer backed Pixmaps.

If possible, I'll also try to integrate gl4es in the wrapper.

[tareksander]

I'm starting to adapt my EGL wrapper to libglvnd now and I'm looking into libxcb to make the X11 platform available. I'll report my progress here.

[tareksander]

Also it can fale to mmap part of file (like in case with ashmem) so it will be better to do it this way

void* adr = mmap(nullptr, desc.width * desc.height * pixelSizeForFormat(desc.format), PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);

Currently you have size 4*4 which does not take into account pixel size (which is 4) so it should be 4*4*4.

No I have 2*2 pixels, so 4, and *4 for the pixel size.

[tareksander]

Ok, it was a C++ error. When an std::thread goes out-of-scope without being terminated, the program gets aborted.

[tareksander]

Oh, and when you try to map a different fd than the right one, apparently the Android emulator restarts.
If that also happens on hardware it would be bad.

[twaik]

You can make something like this:

            std::thread([&] {
                libandroid.AHardwareBuffer_sendHandleToUnixSocket(hb, socks[1]);
                close(socks[1]);
            }).detach();

[tareksander]

You can make something like this:

            std::thread([&] {
                libandroid.AHardwareBuffer_sendHandleToUnixSocket(hb, socks[1]);
                close(socks[1]);
            }).detach();

I just did a join the end of the scope. Also works.

[twaik]

@licy183 Hi. I have an idea how to make virglrenderer-android work much faster than it does now, but it will require patching virglrenderer, mesa and maybe X server. Maybe it would be tough thing.
virglrenderer + mesa (virpipe driver). There is no option to send DMABUF fd for front and back buffers in mesa. Everything is just copied to memfd buffer and sent to virpipe buffer. Also after getting this virpipe simply copies everything into shared memory fragment (yeah, second copy, not very fast) and only after this it can be drawn on screen. But it will be twice faster to bind memfd's fie descriptor to X server with xcb_shm_attach_fd and draw it with xcb_shm_put_image (and that will mean -1 copy, great for perfomance). But it will work only in the case of Unix socket, so old solution should be used in the case if xcb_shm_attach_fd fails.

virgl + mesa (virpipe driver) + X server. I think it should work only after checking only if X server supports DRI3 and Android supports AHardwareBuffer. AHardwareBuffer can be transfered through Unix sockets (preferably socketpair), so it would be much better to set some AHardwareBuffer as back/front buffer and transfered to X server with xcb_dri3_pixmap_from_buffer. After this it will be possible to draw this pixmap with xcb_present_pixmap. DRI3 implementation on X server side will extract AHardwareBuffer from socket and create pixmap from that.

How do you think, can you implement that?

[twaik]

Ok, sorry for interrupting.

[twaik]

Maybe you can implement fake vulkan android wsi? I mean support for vkCreateXcbSurfaceKHR, binding AHardwareBuffers to vulkan images and implement swapchain which contain these images returned by vkGetSwapchainImagesKHR, etc...

[tareksander]

Do you have a build script for Mesa with Zink? I need some way to test it if I implement Vulkan WSI, or just some program packaged for Termux that would use Vulkan X11 WSI?

[twaik]

Do you have a build script for Mesa with Zink?

Maybe you can use one from tur repo.

some program packaged for Termux that would use Vulkan X11 WSI?

Maybe vkcube in vulkan-tools?

[twaik]

Do you have a build script for Mesa with Zink?

https://github.com/termux-user-repository/tur/tree/master/tur/mesa-zink