Why the event on stream is not fired with peerjs and with cordova-plugin-iosrtc when i receive a call? Using PeerJs.

[iometrine]

Hello,

as @hthetiot recommended on the post : "peers/peerjs#705" i create a new post to explain my issue.

Versions

"@angular/common": "~9.1.6",
"@capacitor/ios": "^2.2.1",
"cordova-plugin-ionic": "5.4.7",
"cordova-plugin-iosrtc": "^6.0.18",
xcode : 12.3
peerJs: 1.3.1 

Description

i 'm working with peerjs and iosrtc plugin and i have got a problem on reception with ios, this method on stream is never fired.

call.on('stream', (stream) => {
document.getElementById('camera').srcObject = stream;
});
And this event neither:

call.peerConnection.addEventListener('addstream', (e: any) => {
document.getElementById('camera').srcObject = e.stream;
});
And to finish as i've read on other post, this method doesn't exist anymore today

call.peerConnection.onaddstream = function (stream) { // do something with stream }.

Can you help me ?

[abardik]

@iometrine onaddstream event is deprecated as well as addStream/removeStream methods of RTCPeerConnection (https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection). You should use ontrack event and addTrack/removeTrack methods instead, but I think the latest peerjs should already contain them.

Or you can try to include adapter-latest.js (https://github.com/webrtchacks/adapter), maybe it will help since it has implemented wrappers for many of deprecated WebRTC methods, but this is not the right way to go. Anyway, you should go away from peerjs's 'stream approach' as soon as you can if you want to proceed with add/remove/replace media tracks in your calls.

[flamurhbreznica]

@abardik facing the same issue. It is working in the first time (adding ontrack) but not in the second time and onwards. Any tips ?

Info: we are using Ionic, it is working perfectly in android but not in ios.

[iometrine]

I tried to install webrtchacks, it doesn't work.
I tried to add a listener ontrack and addTRack, it doesn't work.

[flamurhbreznica]

To be more precise:

I have VideoCall System developed through Ionic Cordova with PeerJS, it works properly on Android.
Now that I deployed to iOS from cordova, whenever I make second call, it always set an error: The PluginMediaStream with this id "any id" already exists, and cannot join to calls anymore, also the events from peerjs does not trigger

for example:

peer.on('stream') does not trigger at all, I used this plugin to handle the MediaStreams, but for some reasons, after the first video call it says MediaStream already exists,

How i can I trigger some events if I get a call from peer, onstream, onremove,

I tried to remove streams, tracks, and other stuff after ending video call, but it did not work.

[iometrine]

I think i have almost in the same situation as you but i can't receive the first call.
All works on android and the web as well.
Apparently the problem is known with cordova, I followed the documentation below :
https://github.com/cordova-rtc/cordova-plugin-iosrtc#ios-safari-and-crash-on-websocket-events

I added the <script src="ios-websocket-hack.min.js"></script>
and i added a setTimout()

I ran many test and I don't know what to do now.

[abardik]

@iometrine The latest peerjs has implemented ontrack and addTrack, so you should be ok to use it with iosrtc plugin:
https://github.com/peers/peerjs/blob/cfc37c7988d8ef3d2c1d7b6123562dd2af59defc/lib/negotiator.ts#L140
https://github.com/peers/peerjs/blob/cfc37c7988d8ef3d2c1d7b6123562dd2af59defc/lib/negotiator.ts#L345

Maybe the problem is that you use the example from their homepage:
var getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia;
It should be navigator.mediaDevices.getUserMedia or cordova.plugins.iosrtc.getUserMedia.

Also, you should request user permissions to access iPhone camera/mic:
cordova.plugins.iosrtc.requestPermission.

[abardik]

@flamurhbreznica To properly end a call in peerjs, which stores local and remote streams in its own properties, you should stop and remove all tracks from local stream as well as the local stream itself at the end of each call, and you should do this for both caller and callee, something like this:

call.localStream.getTracks().forEach(function(track) {
	track.stop();
	call.localStream.removeTrack(track);
});
delete call.localStream;
call.localStream = null;

Also, it would be also good if you manually remove closed calls from peer.connections, which does not clean closed data/media connections by itself, at least in earlier versions (check this out). After these cleanings you should be able to make a second call.

[flamurhbreznica]

@abardik we have implemented this but the issue persists -> it works in the first time but not in the second and onwards. Could there be another solution ?

[hthetiot]

thank you @abardik for your help, it works for you with iosrtc and peerJs if i understood properly?

@flamurhbreznica you may want to try this PR, see instructions:

• implement PREFFIX_INVALID_MEDIA_ID pluxin.xml preference to disable UUID preffix for bad media ids via config #647 #648 (comment)

[hthetiot]

note: You can also try master that have totally new WebRTC and major changes and will be released as 8.0.0

[hthetiot]

@iometrine please provide PeerJs version, Steps to reproduce, etc.. also you did not respect the required information:

https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/.github/ISSUE_TEMPLATE.md#you-must-read-first

Note: If the checkboxes above are not checked (which you do after the issue is posted), the issue will be closed, removing this checkbox will result in automatic closed issue.

[hthetiot]

Would be interesting to add peerJS sample here https://github.com/cordova-rtc/cordova-plugin-iosrtc-sample/tree/master/www/js to assist peerJS support and reproduce.

[abardik]

@hthetiot Yes, it works. But I added a proper cleaning as described above and custom in-call renegotiation logic, because peerjs did not provide it at that time (I don't know about the latest versions, but it seams like still no). Also, for binary data channels messages should be converted to Uint8Array.

[flamurhbreznica]

@abardik could you please tell us how you handled the in-call renegotiation logic since we have implemented the rest of what you have suggested but it is still not working.

[abardik]

Would be interesting to add peerJS sample here https://github.com/cordova-rtc/cordova-plugin-iosrtc-sample/tree/master/www/js to assist peerJS support and reproduce.

@hthetiot I'll try to make an example, but can't guarantee any timeframe, sorry.

[flamurhbreznica]

thank you @abardik for your help, it works for you with iosrtc and peerJs if i understood properly?

@flamurhbreznica you may want to try this PR, see instructions:

• #648 (comment)

@hthetiot we just tried this too, still the same. Working in the first time but not in the second and onwards.

[abardik]

@flamurhbreznica Since you did not provide your code, I just can say that the second call is all about a proper cleaning, just make sure you stopped and removed all tracks/streams/connections. But in-call renegotiation is much more complicated. Peerjs doesn't expect another OFFER inside an already established call as well as doesn't provide an ability to make an OFFER after connection is already established saying something like "Are you trying to answer the call twice?". Also, there is no senders/transceivers in peerjs, so no modern renegotiation from the box.

[fehmicitaku]

@abardik

`this.peer.on('call', async call => {
navigator.mediaDevices.getUserMedia({
audio: true,
video: true,
})

            .catch((err) => {
                    console.error('[Error] Not able to retrieve user media:', err);
                    return null;
            }).then(async (stream: MediaStream | null) =>{

                    this.mystream = stream

                    this.incomingCall = call;

                    const modal = await this.modalController.create({
                        component: IncomingCallComponent,
                        showBackdrop: true,
                        componentProps: {
                            call: call,
                            stream: this.mystream
                        }
                    });
                    modal.onDidDismiss().then(async (data:any) => {
                        if(data.data.accepted) {
                            const callModal = await this.modalController.create({
                                component: PeercallComponent,
                                showBackdrop: true,
                                componentProps: {
                                    call: this.incomingCall,
                                    stream: this.mystream,
                                }
                            });
                            return await callModal.present();
                        }
                    });
                    this.loaderIncoming.dismiss();
                    return await modal.present();
            })
 });`

FYI we haver this Incoming Call Component, as a popup screen to answer/decline the call, then if he accept/answer the call, we move to PeerCallComponent, where we show both of streams (caller & receiver), this works fine on the first call, and works properly every time on Android, but it fails on iOS since we got that error: MediaStream already Exists

as for the end-call function we use this one:

stream.getTracks().forEach(track => { stream.removeTrack(track); track.stop(); });

[abardik]

@fehmicitaku When did you get this error? console.trace please. Also, try to release this.mystream and this.incomingCall as well as remove the closed call from peer.connections if it is still there.

[fehmicitaku]

@fehmicitaku When did you get this error? console.trace please. Also, try to release this.mystream and this.incomingCall as well as remove the closed call from peer.connections if it is still there.

Im not getting errors inside my code, the errors/logs are coming from iosrtc plugin which says: the PluginMediaStream with id "2728191iq9q9q91" already exists, this is all what i can see on xcode console tab, which comes from iosrtc, theres no error from my code, i tried to remove all calls and all tracks, but still the same unfortunately

[hthetiot]

the PluginMediaStream with id "2728191iq9q9q91" already exists

This logs Might be not be what you think it is, i would not pay attention to that too much.

[fehmicitaku]

@fehmicitaku When did you get this error? console.trace please. Also, try to release this.mystream and this.incomingCall as well as remove the closed call from peer.connections if it is still there.

@abardik how do i check the peer connections if theres a call?

[abardik]

@abardik how do i check the peer connections if theres a call?

Call is a MediaConnection. peer.connections includes both DataConnections and MediaConnections. Every call is placed in peer.connections and you can find it there something like this: peer.connections[remotePeerId].indexOf(call)

[iometrine]

@hthetiot i'm using peerjs 1.3.1 (stable version)

[iometrine]

@fehmicitaku how did you do because the first call doesn't work for me.
which version of peerjs do you use ?
did you include "ios-websocket-hack.min.js" on your index.html ?
Do you use https://github.com/webrtchacks/adapter ?
Can you show me the listener you use to trap the call for the receiver (onTrack or onstream).

I try all the event below:

    
    setTimeout(() => {
    this.mediaConnection.answer(this.myStream);
 
    this.mediaConnection.peerConnection.addEventListener('addstream', (e: any) => {
      
      this.partnerEl.srcObject = e.stream;
    });
    this.mediaConnection.peerConnection.addEventListener('addtrack', (e: any) => {
      this.partnerEl.srcObject = e.stream;
    });

    this.mediaConnection.peerConnection.addEventListener('ontrack', (e: any) => {
      
      this.partnerEl.srcObject = e.stream;
    });
      
    this.mediaConnection.on('stream', function(stream)  {
      this.partnerEl.srcObject = stream;
    });
    }, 500);

About your problem, why don't you close the peer connection at the end and you create a new instance of your peer ?

[flamurhbreznica]

@iometrine We tried the adapter but it did not fix our issue. We are using version 2.1 try updating maybe it will work for you.

@abardik tried it but still not working. In other words what is happening is that we are always able to call the other device. But whenever the other device calls us it works only in the first time (after we login in the app) never the second. The other device can see us through the video but we cannot see them or ourselves.

[iometrine]

2.1 ? the last version is 1.3.2
https://github.com/peers/peerjs/releases

[flamurhbreznica]

2.1 ? the last version is 1.3.2
https://github.com/peers/peerjs/releases

Typo 1.2***

[iometrine]

So I have a newer version than you.
Can you show me the listener you use to trap the call for the receiver (onTrack or onstream).

[hthetiot]

We are using version 2.1 try updating maybe it will work for you.

@flamurhbreznica
Using the latest version 1.3.2 is what you should try first before anything.
See changes here: https://github.com/peers/peerjs/blob/master/changelog.md#132-2021-03-11

[hthetiot]

If someone take the time to implement an peerjs sample via index-peerjs.js inside the sample here:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc-sample/tree/master/www/js

It may help a lot to debug.

[hthetiot]

I do not reproduce the issue using cordova-plugin-sample with peerjs 1.3.1, a can make multiple call without issues.
I have a video but i have connexion issue to upload, will upload later here for proof.

index-peerjs.js:

function TestRTCPeerConnection() {

	return loadScript('https://unpkg.com/[email protected]/dist/socket.io.js').then(function () {
		return loadScript('https://unpkg.com/[email protected]/dist/peerjs.min.js').then(function () {
			return joinRoom(localStream);
		});
	});
}

const peers = {};
const ROOM_ID = 'test';
//const HOSTNAME = '192.168.1.13';
const HOSTNAME = '';

function joinRoom(localStream) {

	const socket = io(HOSTNAME + ':3000/');
	const myPeer = new Peer(undefined, {
		host: HOSTNAME || '/',
		port: '3001'
	})

	myPeer.on('call', call => {
		call.answer(localStream)
		call.on('stream', userVideoStream => {
		  TestControlsOutgoingCall();
		  TestSetPeerStream(userVideoStream);
		})
	})

	socket.on('user-connected', userId => {
		const call = myPeer.call(userId, localStream);

		call.on('stream', userVideoStream => {
			TestControlsOutgoingCall();
            TestSetPeerStream(userVideoStream);
		})

		call.on('close', () => {
			TestControlsClosingCall();
		})

		peers[userId] = call;
	})

	socket.on('user-disconnected', userId => {
	  if (peers[userId]) peers[userId].close()
	})

	myPeer.on('open', id => {
	  socket.emit('join-room', ROOM_ID, id)
	})
}

server.js

const express = require('express')
const app = express()
const server = require('http').Server(app)
const io = require('socket.io')(server)
const { v4: uuidV4 } = require('uuid')
const { PeerServer } = require('peer');

app.set('view engine', 'ejs')
app.use(express.static('public'))

app.get('/', (req, res) => {
  res.redirect(`/${uuidV4()}`)
})

app.get('/:room', (req, res) => {
  res.render('room', { roomId: req.params.room })
})

io.on('connection', socket => {
  socket.on('join-room', (roomId, userId) => {
    socket.join(roomId)
    socket.to(roomId).broadcast.emit('user-connected', userId)

    socket.on('disconnect', () => {
      socket.to(roomId).broadcast.emit('user-disconnected', userId)
    })
  })
})

server.listen(3000)

const peerServer = PeerServer({ port: 3001, path: '/' });

[iometrine]

Hi all,

Sorry for my late response, I did a lot of testing to try and figure out the problem with your various recommendations and help.
I created a project from scratch to reproduce the problem as @hthetiot recommends to me.
But when I finished, the POC, the project is running and I couldn't reproduce the problem.

Finally after looking everywhere, I noticed that the plugin version was different.
my poc used 6.0.20 and my project 6.0.18.
So I changed the version and now the event is triggered when I get a call.

For information, I share the POC if it can help anyone.
https://github.com/iometrine/ionic-iosrtc-peerjs

[hthetiot]

Wonderful @iometrine

@flamurhbreznica you know what you have to do now :)

[hthetiot]

@iometrine i forked your project here https://github.com/cordova-rtc/ionic-iosrtc-peerjs

Can be useful to the community.

[flamurhbreznica]

@iometrine what is POC

[hthetiot]

@iometrine what is POC

POC mean "Proff of concept" see Wikipedia https://en.wikipedia.org/wiki/Proof_of_concept