surfaces that configured to physical camera just got one frame after SetRepeatingRequest

[epiciskandar]

SPECIFIC ISSUE ENCOUNTERED

My purpose: use ARCore to get camera 3D position and posture data, and recording individual physical back camera, all at the same time.

When I tried using ARCore with Camera2 api which enables additional individual physical camera recording/preview, only the main(logical) camera got images that could be rendering, not those physical cameras.

If I call the Camera2 API doing the same thing, it works.

VERSIONS USED

• Android Studio: Android Studio Koala | 2024.1.1 Patch 2
• ARCore SDK for Android: 1.45.0
• Device manufacturer, model, and O/S: Samsung Galaxy S22 Ultra, Android 13 (also tried on other devices such as Pixel 7)
• Google Play Services for AR (ARCore): 1.45.242050293
  On Windows, use: adb shell pm dump com.google.ar.core | findstr /i "packages: versionName"
  On macOS, use: adb shell pm dump com.google.ar.core | egrep -i versionName\|packages:
• Output of adb shell getprop ro.build.fingerprint: samsung/b0qzcx/b0q:13/TP1A.220624.014/S9080ZCU4CWH1:user/release-keys

STEPS TO REPRODUCE THE ISSUE

sorry for inconvenient providing full demo

1. with GLSurfaceView, when onSurfaceCreated, create session with shared camera Session(this, setOf(Session.Feature.SHARED_CAMERA))
2. configure shared session with auto focus, DepthMode.AUTOMATIC, and UpdateMode.LATEST_CAMERA_IMAGE
3. call cameraManager.openCamera with wrapped callback
4. in CameraDevice.StateCallback.onOpened:

• call setCameraTextureName for rendering main video stream openGL rendering, and configure a list of List<OutputConfiguration> with surfaces from sharedCamera.arCoreSurfaces, alongside with surfaces that comes from jetpack compose UI constructing which configured with specified physical camera by calling OutputConfiguration.setPhysicalCameraId
• store surfaces from list above, for further repeating capture request
• cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_RECORD)
• SessionConfiguration(SessionConfiguration.SESSION_REGULAR, outputConfigs, ...) with wrapped StateCallback

5. in CameraCaptureSession.StateCallback.onConfigured, call session.setRepeatingRequest with surfaces saved previously, it is sure that createCaptureRequest and setRepeatingRequest have the same target/surface config.

Now, the logical camera render to GLSurfaceView corretly, but SurfaceViews that should showing physical camera preview just got one frame.

video below:

scr_rec.mp4

in the video, the beginning half shows calling ARCore and Camera2, notice that sub previews only got updated one frame. at the end, video shows how the preview should be shown to views(and currently main preview have bug, not related here) if only using Camera2.

WORKAROUNDS (IF ANY)

ADDITIONAL COMMENTS

The ARCore update is driven by sharedSession.update() at GLSurfaceView.Renderer.onDrawFrame, and I got camera position data from frame.camera.pose

[epiciskandar]

This problem could be fixed by "double commit" like this:

val callback = object : CameraCaptureSession.StateCallback() {
    override fun onConfigured(session: CameraCaptureSession) {
        [email protected] = session
        val rr = object: Runnable {
            override fun run() {
                session.setRepeatingRequest(
                    requestBuilder.build(),
                    cameraCaptureCallback,
                    cameraHandler
                )
            }
        }
        cameraHandler.postDelayed(rr, 0)
        cameraHandler.postDelayed(rr, 100)  // could not be too small, and must post two times
    }
}

but I don't know why

edit:
This only works on Samsung, on the Pixel 7, double commit will stop main(logical) camera receiving data, and got this error log:

2024-08-23 16:33:44.464  7194-7376 gesture         com.gesture             E  [SurfaceTexture-0-7194-0] detachFromContext: SurfaceTexture is not attached to a GL context
2024-08-23 16:33:44.464  7194-7376  ArCore-TextureStore     com.gesture             E  attachTexImage: calling detachFromGLContext failed:Error during detachFromGLContext (see logcat for details)

But add delay time will make it works on Pixel 7 too, like this:

        cameraHandler.postDelayed(rr, 100)
        cameraHandler.postDelayed(rr, 300)