ScreenSource problem

[DenisStadnichenko]

Hi, thanks so much again for your hard work and wonderful library. I have a problem with screen capture, when using ScreenSource the screen is captured by my app, if I minimize the app to the background, the phone screen is not captured and transmitted for broadcast. The app is in pip mode.

Can you tell me how the capture should work, can screen capture occur after the application is minimized to the backgroud?

My idea is to stream games, but when I minimize the app I don't get a picture of the game, only a black screen of my app

@AndroidEntryPoint
class NewRTMPService : Service(), ConnectChecker {

    companion object {
        private const val TAG = "DisplayService"
        private const val channelId = "TestChannel"
        const val notifyId = 123456
        var INSTANCE: NewRTMPService? = null
    }

    private var notificationManager: NotificationManager? = null
    private lateinit var genericStream: GenericStream

    private var mediaProjection: MediaProjection? = null
    private val mediaProjectionManager: MediaProjectionManager by lazy {
        applicationContext.getSystemService(MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
    }

    private var callback: ConnectChecker? = null
    private val width = 640
    private val height = 480
    private val vBitrate = 1200 * 1000
    private var rotation = 0 //0 for landscape or 90 for portrait
    private val sampleRate = 32000
    private val isStereo = true
    private val aBitrate = 128 * 1000
    private var prepared = false
    private var isFlip = false
    private var isMuteAudio = false

    override fun onCreate() {
        super.onCreate()
        Timber.tag(TAG).i("RTP Display service create")

//        notificationManager = getSystemService(Context.NOTIFICATION_SERVICE) as NotificationManager
//        val channel =
//            NotificationChannel(channelId, channelId, NotificationManager.IMPORTANCE_HIGH)
//        notificationManager?.createNotificationChannel(channel)

        // createNotificationChannel()
        //     startForeground()


        // doForegroundThings()
        // showNotification("Create Test service")
        genericStream =
            GenericStream(baseContext, this, CameraXSource(baseContext), MicrophoneSource()).apply {
                //This is important to keep a constant fps because media projection only produce fps if the screen change
                getGlInterface().apply {
                    //  setForceRender(true, 15)
                    autoHandleOrientation = true
                    setForceRender(true, 25)

                    //  forceCodecType(CodecUtil.CodecType.HARDWARE, CodecUtil.CodecType.HARDWARE)
                }
            }

        INSTANCE = this
    }

    private fun createNotificationChannel() {
        notificationManager = getSystemService(Context.NOTIFICATION_SERVICE) as NotificationManager
        val name = resources.getString(R.string.channel_name)
        val descriptionText = "T Live Multistreaming"
        val importance = NotificationManager.IMPORTANCE_DEFAULT
        val channel = NotificationChannel(channelId, name, importance).apply {
            description = descriptionText
        }
        notificationManager?.createNotificationChannel(channel)
    }

    fun doForegroundThings() {
        Timber.tag(TAG).i("RTP Display service doForegroundThings")
        createNotificationChannel()
        val notificationIntent = Intent(this, MainActivity::class.java)
        val pendingIntent =
            PendingIntent.getActivity(
                this,
                0,
                notificationIntent,
                PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE
            )
        val builder = NotificationCompat.Builder(this, channelId)
            .setSmallIcon(R.drawable.ic_logo_2)
            .setContentTitle("Test")
            .setContentText("Started")
            .setContentIntent(pendingIntent)
            .setPriority(NotificationCompat.PRIORITY_HIGH)
            .setVisibility(NotificationCompat.VISIBILITY_PUBLIC)
            .setPriority(NotificationCompat.PRIORITY_DEFAULT)

        val notification = builder.build()
        with(NotificationManagerCompat.from(this)) {
            // notificationId is a unique int for each notification that you must define
            if (ActivityCompat.checkSelfPermission(
                    this@NewRTMPService,
                    Manifest.permission.POST_NOTIFICATIONS
                ) != PackageManager.PERMISSION_GRANTED
            ) {
                return
            }
            notify(notifyId, notification)
        }

        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) {
            startForeground(
                notifyId,
                notification,
                ServiceInfo.FOREGROUND_SERVICE_TYPE_CAMERA or ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION
            )
        } else {
            startForeground(notifyId, notification)

        }

    }


    private fun showNotification(text: String) {
        val intent = Intent(this, MainActivity::class.java).apply {
            flags = FLAG_ACTIVITY_CLEAR_TOP or FLAG_ACTIVITY_SINGLE_TOP
        }
        val pendingIntent: PendingIntent = PendingIntent.getActivity(
            this,
            0,
            intent,
            PendingIntent.FLAG_MUTABLE or PendingIntent.FLAG_UPDATE_CURRENT
        )

        val stopIntent = Intent(this, NewRTMPService::class.java)
        stopIntent.action = "STOP_SERVICE"
        val stopPendingIntent = PendingIntent.getService(
            this,
            0,
            stopIntent,
            PendingIntent.FLAG_IMMUTABLE or PendingIntent.FLAG_CANCEL_CURRENT
        )

        val notification = NotificationCompat.Builder(applicationContext, channelId)
            .setSmallIcon(R.drawable.ic_logo_2)
            .setContentTitle("TestApp")
            .setContentText(text)
            .setPriority(NotificationCompat.PRIORITY_HIGH)
            .setVisibility(NotificationCompat.VISIBILITY_PUBLIC)
            .setContentIntent(pendingIntent)
            .setDeleteIntent(stopPendingIntent)
            .setAutoCancel(true)
            .build()

        ServiceCompat.startForeground(
            this,
            notifyId,
            notification,
            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) {
                ServiceInfo.FOREGROUND_SERVICE_TYPE_CAMERA or
                        ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION

            } else {
                0
            }
        )

        notificationManager?.notify(notifyId, notification)
        startForeground(notifyId, notification)
    }


    override fun onBind(p0: Intent?): IBinder? {
        return null
    }

    override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
        Timber.tag(TAG).i("RTP Display service started")
        //  doForegroundThings()
        //   showNotification("Start Test service")
        return START_STICKY
    }

    fun getStreamClient() = genericStream.getStreamClient()

    fun prepare(
        width: Int,
        height: Int,
        bitrate: Int,
        fps: Int,
        iFrameInterval: Int

    ): Boolean {
        Timber.tag(TAG).i("prepare$prepared")
        prepared = try {
            prepareVideo(width, height, bitrate, fps, iFrameInterval) &&
                    prepareAudio()
        } catch (e: IllegalArgumentException) {
            false
        }

        Timber.tag(TAG).i("prepare$prepared")


        if (!prepared) {
            Timber.tag(TAG).i("Audio or Video configuration failed")
        }

        return prepared
    }

    private fun prepareVideo(
        width: Int, height: Int, bitrate: Int, fps: Int = 30, iFrameInterval: Int = 2,
        rotation: Int = 0, profile: Int = -1, level: Int = -1
    ): Boolean {
        Timber.tag("RTMPService.TAG")
            .i("prepare already: width: $width, height: $height, bitrate: $bitrate, fps: $fps, iFrameInterval: $iFrameInterval")

        return genericStream.prepareVideo(
            width = width,
            height = height,
            bitrate = bitrate * 1000,
            rotation = rotation,
            fps = fps,
            iFrameInterval = iFrameInterval
        )
    }

    fun prepareAudio(): Boolean {
        return genericStream.prepareAudio(sampleRate, isStereo, aBitrate)
    }

    fun sendIntent(): Intent {
        return mediaProjectionManager.createScreenCaptureIntent()
    }

    fun prepareLiveScreenStream(resultCode: Int, data: Intent): Boolean {
        //keepAliveTrick()

        // mediaProjection?.stop()

        val mediaProjection = mediaProjectionManager.getMediaProjection(resultCode, data)
        this.mediaProjection = mediaProjection
        val screenSource = ScreenSource(
            this,
            mediaProjection,
            mediaProjectionCallback = object : MediaProjection.Callback() {
                override fun onStop() {
                    super.onStop()

                    Timber.i("STOOOP")
                }
            },
            virtualDisplayCallback = object : VirtualDisplay.Callback() {
                override fun onStopped() {
                    super.onStopped()

                    Timber.i("STOOOP 3")
                }
            })
        // screenSource.rotation = 90
        // screenSource.height = 1024
        // screenSource.width = 780
        return try {
            //screenSource.resize(1024,780)
            //ScreenSource need use always setCameraOrientation(0) because the MediaProjection handle orientation.
            //You also need remove autoHandleOrientation if you are using it.
            //You need to call it after prepareVideo to override the default value.
            //   genericStream.getGlInterface().setCameraOrientation(0)
            genericStream.changeVideoSource(screenSource)
//genericStream.videoSource.width = 780
//genericStream.videoSource.height = 1024

            genericStream.changeAudioSource(MicrophoneSource())



            true
        } catch (ignored: IllegalArgumentException) {
            false
        }
    }


    fun startForeground() {
        doForegroundThings()
        showNotification("Stream prepare")
    }


    fun isStreaming(): Boolean {
        return genericStream.isStreaming
    }

    fun isRecording(): Boolean {
        return genericStream.isRecording
    }

    fun isOnPreview(): Boolean {
        return genericStream.isOnPreview
    }

    fun startPreview(surfaceView: SurfaceView) {
        genericStream.startPreview(surfaceView)
    }

    fun stopPreview() {
        genericStream.stopPreview()
    }

    fun setPreviewResolution(
        width: Int,
        height: Int
    ) {
        genericStream.getGlInterface().setPreviewResolution(width, height)
    }

    fun stopStream() {
        if (genericStream.isStreaming) {
            Timber.tag(TAG).i("RTP Display service stop")
            genericStream.stopStream()
            mediaProjection?.stop()
            notificationManager?.cancel(notifyId)
            stopForeground(STOP_FOREGROUND_REMOVE)
        }
    }

    fun setCallback(connectChecker: ConnectChecker?) {
        callback = connectChecker
    }

    override fun onDestroy() {
        super.onDestroy()
        Timber.tag(TAG).i("RTP Display service destroy")
        stopStream()
        genericStream.release()
        mediaProjection?.stop()
        mediaProjection = null

        notificationManager?.cancel(notifyId)
        stopSelf()
        INSTANCE = null
    }

    fun startStream(endpoint: String) {
        if (!genericStream.isStreaming) genericStream.startStream(endpoint)
    }

    override fun onConnectionStarted(url: String) {
        callback?.onConnectionStarted(url)
        doForegroundThings()
        showNotification("Stream started")
    }

    override fun onConnectionSuccess() {
        callback?.onConnectionSuccess()
    }

    override fun onNewBitrate(bitrate: Long) {
        callback?.onNewBitrate(bitrate)
    }

    override fun onConnectionFailed(reason: String) {
        callback?.onConnectionFailed(reason)
    }

    override fun onDisconnect() {
        callback?.onDisconnect()
    }

    override fun onAuthError() {
        callback?.onAuthError()
    }

    override fun onAuthSuccess() {
        callback?.onAuthSuccess()
    }
}

[pedroSG94]

Hello,

Did you test using the app example with screen example?
Also, remember that depend of the Android version when the dialog appear, the dialog ask you if you want record your app or all the screen. You should select all the screen

[DenisStadnichenko]

Hi, I found an issue that was causing the app to capture only the app screen and not the whole phone.

Initially I run GenericStream with CameraXSource to be able to stream from the device camera, but I want to be able to stream from the screen.

genericStream = GenericStream(baseContext, this, CameraXSource(baseContext), MicrophoneSource())

Can you please tell me how to update videoSource in GenericStream correctly? I tried to set NoVideoSource before installing ScreenSource, but it didn't work. I think I need to completely recreate GenericStream with default NoVideoSource and then replace it with ScreenSource.

Is this correct or is there another way?

[pedroSG94]

Do you have this problem using the library app example? can you tell me Android version of your device?

[DenisStadnichenko]

Device pixel 3a xl. 12 android
I used your code from the example to create my backend service that initially streams video from the camera, I tried to replace VideoSource with ScreenSourse and my problem was that VirtualDisplay only captured the application screen, if you use NoVideoSource (like in your example ScreenActivity) when initializing GenericStream then everything works

[pedroSG94]

Hello,

If I go here:
https://github.com/pedroSG94/RootEncoder/blob/master/app/src/main/java/com/pedro/streamer/screen/ScreenService.kt#L87
And I replace NoVideoSource to CameraXSource, it is working for me without problems.

Can you tell me the code that I need replace to reproduce the error?

[DenisStadnichenko]

I have this problem in my background service, I took the code from old examples with backgroundexample,
It was probably somewhere in these links, can't find it in the repository

#1369

I start by creating a GenericStream with CameraXSource and then replace it with ScreenSource

I assume that I need to create 2 separate backgroud services and run the necessary one depending on the task.

[pedroSG94]

Hello,

You can use the same service for it. Using GenericStream you can replace the VideoSource on fly.
The issue that you shared is not using GenericStream, it is using RtmpCamera2 that works totally different if you want go to background.

Do you have a full code example and steps to do to reproduce the error?

[DenisStadnichenko]

Hi, after researching I found that stopping preview (genericStream.stopPreview()) solves all my problem. I assume that since I am using the camera first, the application when capturing the video has focus on it, after genericStream.stopPreview() everything is fine )

The branch can be closed.

Thanks for your help and for the great library.
Best regards!

[pedroSG94]

Closing as solved

[pedroSG94]

Closed