Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ICE connectivity check failures #3738

Closed
ghost opened this issue Dec 26, 2018 · 21 comments
Closed

ICE connectivity check failures #3738

ghost opened this issue Dec 26, 2018 · 21 comments
Labels
wontfix Issue won't be fixed

Comments

@ghost
Copy link

ghost commented Dec 26, 2018

Description


In a two-server environment, one host is running jitsi-meet (from a recently jitsi nightly deb package), with some small configuration modifications detailed below, and another host is running webrtc-streamer (with a small fix applied as per #688).

The streamer client appears to join the room and the browser client expands a full black screen to commence video output, but connectivity from the streamer client drops out, reproducibly.

There are some existing reports (both in the Jitsi community forum, and under the https://github.com/jitsi repo issue trackers) of similar issues and most of them point towards issues with NAT and ICE configuration - in this case the server is configured with near-default settings. The modifications from default are:

  • /etc/apache2/sites-enabled/localhost.conf: Header set Access-Control-Allow-Origin "*" added to allow cross-domain JS requests
  • /etc/prosody/prosody.cfg.lua: cross_domain_bosh = true added to allow cross-domain JS requests
  • /etc/jitsi/meet/localhost-config.js - config.p2p.enabled: false - configured since the environment is intended for a minimum of three participants (streamer and at least two human discussion partners)

No firewalling is present on the jitsi-meet server. Attempts to use each of the following configuration keys (often in combination) in /etc/jitsi/videobridge/sip-communicator.properties have been attempted but without any success so far:

  • org.jitsi.videobridge.TCP_HARVESTER_PORT
  • org.jitsi.videobridge.TCP_HARVESTER_MAPPED_PORT
  • org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS
  • org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS

The following similar reading references have been found:

Current behavior


  • Human client joins channel test
  • Streamer client joins channel test for stream named Example
  • Human client observes participant named Example joining the room
  • Human client observes connectivity issues (see screenshot) for participant Example, without receiving any video/audio from them

connectivity-issues

Expected Behavior


  • Participant Example is able to join and maintain a connection to the room

Possible Solution


This seems potentially related to ICE and client connectivity checks. The following warnings appear in the videobridge logs:

JVB 2018-12-26 01:05:46.475 INFO: [266] org.ice4j.ice.ConnectivityCheckClient.log() Error response for pair: 192.168.2.126:10000/udp/host -> 192.168.2.126:41487/udp/prflx (stream.RTP), failing.  Code = 401(class=4; number=1)

The error code 401 from these logs appears to be retrieved in ice4 and documented in RFC 3489.

In the client, which builds upon the Google webrtc project, warnings of the following format appear:

[042:060] [12565] (port.cc:614): Port[e807f890:video:1:0:local:Net[wlan0:192.168.2.x/24:Unknown:id=3]]: Received STUN request with bad local username gZhD from 192.168.2.x:10000

Steps to reproduce


  • Configure two hosts; one jitsi-meet, one webrtc-streamer with packages and versions documented here
  • Apply documented configuration changes from this issue
  • Join a channel as a human client
  • Begin streaming to the same channel via xmppvideostream.html in webrtc-streamer

NB: This same behavior can be observed by running webrtc-streamer locally against the cloud-hosted https://meet.jit.si instance - however server logs are not available in that case.

Environment details


@ghost
Copy link
Author

ghost commented Dec 26, 2018

Attaching jvb.log

@ghost
Copy link
Author

ghost commented Dec 27, 2018

https://community.jitsi.org/t/jvb-how-to-listen-to-port-443/15017 also looks relevant, but seems to suggest that setting up two separate machines is required - one for JVB and one for the jitsi-meet webserver.

Is there a convenient way to have everything work 'out of the box' on a single host, or with minor configuration changes? (it'd seem beneficial if possible - allowing new customers to onboard more easily and saving support overhead)

Until discovering the root cause I'm not entirely clear why single-host seems challenging (or if that indeed is the problem I'm hitting here)

@damencho
Copy link
Member

Is your jvb running behind NAT?

@rasos
Copy link
Contributor

rasos commented Dec 27, 2018

The standard jitsi meet set-up on one machine does not allow to harvest TCP 443, it falls back to 4443 as 443 is being used by nginx. This may not affect your usage scenario. Making use of TCP 443 for media transport is supposed to be a "last resort" in case a user is sitting behind a very restricted firewall.

For making use of TCP 443 for media transport you either split signalling (=jicofo+prosody+nginx) and videobridge on two servers, so that the latter also can bind to a 443 port OR you try to replace nginx with jetty web server - however we have not been able to make that work https://git.fairkom.net/hosting/fairmeeting/issues/17

@ghost
Copy link
Author

ghost commented Dec 27, 2018

Thanks @damencho @rasos for your responses.

In this case there is no NAT involved - the jitsi-meet server is listening with apache2 on 443 (using the localhost.conf bundled with the jitsi-meet deb package, with an extra line to enable CORS requests from the client).

In the jvb.log attached earlier I'm trying org.jitsi.videobridge.TCP_HARVESTER_MAPPED_PORT=443 and org.jitsi.videobridge.TCP_HARVESTER_PORT=4443 although I haven't changed any configuration elsewhere (i.e. no firewall rules or apache config changes), so I'm not sure if that's correct.

I'd like to get this working on a single host if possible, partly because I'd like to understand what's required for that, but thank you for the ticket reference @rasos, I'll try split signalling if I can't make further progress.

@damencho
Copy link
Member

I don't get why you need to touch these:
org.jitsi.videobridge.TCP_HARVESTER_PORT
org.jitsi.videobridge.TCP_HARVESTER_MAPPED_PORT
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS

What is your idea?

So NAT_HARVESTER_LOCAL_ADDRESS and NAT_HARVESTER_PUBLIC_ADDRESS, need to be set if jvb does not have locally configured its public address, cause jvb needs to add it in signaling.
TCP_HARVESTER_PORT is to change the port used to listen for TCP connections, which by default is either 443 or 4443 depends on the webserver used.
TCP_HARVESTER_MAPPED_PORT is used to change the TCP port announced in the signaling, but the local used port for listening is not changed, this is normally used with some NAP mapping or iptables forwarding.

@ghost
Copy link
Author

ghost commented Dec 27, 2018

@damencho I'm fairly confident that the NAT settings should not be required; in this issue I'm reproducing everything within a local network, and JVB starts up and listens on an IP and port which the client can access:

# ps faux|grep jvb|more
root      8777  0.0  0.0   9384   784 pts/3    S+   08:18   0:00              |             
      \_ tail -F /var/log/jitsi/jicofo.log /var/log/jitsi/jvb.log /var/log/prosody/prosody.l
og
root      9162  0.0  0.0  16308  1096 pts/4    S+   08:19   0:00                            
      \_ grep --color=auto jvb
jvb       8608  6.2  3.3 6801840 128608 ?      Ssl  08:18   0:05 java -Xmx3072m -XX:+HeapDum
pOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jitsi-videobridge/l
ib/native/linux-64 -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.si
p.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=
/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties -cp
 /usr/share/jitsi-videobridge/jitsi-videobridge.jar:/usr/share/jitsi-videobridge/lib/agafua-
syslog-0.4.jar:/usr/share/jitsi-videobridge/lib/annotations-15.0.jar:/usr/share/jitsi-videob
ridge/lib/bccontrib-1.0.jar:/usr/share/jitsi-videobridge/lib/bcpkix-jdk15on-1.54.jar:/usr/sh
are/jitsi-videobridge/lib/bcprov-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/callstats

root@point:~# netstat -lnp|grep 8608
tcp6       0      0 192.168.2.126:443       :::*                    LISTEN      8608/java           
udp6       0      0 192.168.2.126:10000     :::*                                8608/java           
udp6       0      0 :::55942                :::*                                8608/java

My plan / thinking with the overriden TCP harvesting settings was that perhaps the client wasn't aware of the 443 -> 4443 fallback and that a mapping could help. On second thoughts I don't think that should be required. Attached here is a jvb.log where I've removed those settings.

jvb.log

Next I will try the NAT_HARVESTER_* settings, although again - all of this is network-local on a single host, so I don't believe any NAT tricks should be required here.

@ghost
Copy link
Author

ghost commented Dec 27, 2018

@damencho My mistake; the network paste above was from a previous execution (where I had moved apache2 to 4443; this didn't work either). Here it is again from the configuration matching https://github.com/jitsi/jitsi-meet/files/2712878/jvb.log

# ps faux|grep jvb|more
root     12201  0.0  0.0  16308  1044 pts/3    S+   08:31   0:00              |                   \_ grep --color=auto jvb
jvb      11798  6.7  3.4 6801840 131108 ?      Ssl  08:30   0:05 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jitsi
-videobridge/lib/native/linux-64 -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.commu
nicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties -cp /usr/share/jitsi-videobridge/jitsi-videobridge.ja
r:/usr/share/jitsi-videobridge/lib/agafua-syslog-0.4.jar:/usr/share/jitsi-videobridge/lib/annotations-15.0.jar:/usr/share/jitsi-videobridge/lib/bccontrib-1.0.jar:/usr/sha
re/jitsi-videobridge/lib/bcpkix-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/bcprov-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/callstats-java-sdk-5.0.0.jar:/u
sr/share/jitsi-videobridge/lib/cglib-nodep-2.2.2.jar:/usr/share/jitsi-videobridge/lib/commons-codec-1.6.jar:/usr/share/jitsi-videobridge/lib/commons-lang3-3.1.jar:/usr/sh
are/jitsi-videobridge/lib/commons-logging-1.2.jar:/usr/share/jitsi-videobridge/lib/concurrentlinkedhashmap-lru-1.0_jdk5.jar:/usr/share/jitsi-videobridge/lib/core-2.0.1.ja
r:/usr/share/jitsi-videobridge/lib/dnsjava-2.1.7.jar:/usr/share/jitsi-videobridge/lib/dom4j-1.6.1.jar:/usr/share/jitsi-videobridge/lib/fmj-1.0-20161207.221530-23.jar:/usr
/share/jitsi-videobridge/lib/gson-2.3.1.jar:/usr/share/jitsi-videobridge/lib/guava-15.0.jar:/usr/share/jitsi-videobridge/lib/httpclient-4.4.jar:/usr/share/jitsi-videobrid
ge/lib/httpcore-4.4.jar:/usr/share/jitsi-videobridge/lib/ice4j-2.0.0-SNAPSHOT.jar:/usr/share/jitsi-videobridge/lib/jain-sip-ri-ossonly-1.2.98c7f8c-jitsi-oss1.jar:/usr/sha
re/jitsi-videobridge/lib/java-dogstatsd-client-2.5.jar:/usr/share/jitsi-videobridge/lib/java-sdp-nist-bridge-1.1.jar:/usr/share/jitsi-videobridge/lib/javax.servlet-api-3.
1.0.jar:/usr/share/jitsi-videobridge/lib/jbosh-0.9.0.jar:/usr/share/jitsi-videobridge/lib/jcip-annotations-1.0.jar:/usr/share/jitsi-videobridge/lib/jcl-core-2.8.jar:/usr/
share/jitsi-videobridge/lib/jetty-client-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-continuation-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/je
tty-http-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-io-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-proxy-9.2.10.v20150310.jar:/usr/share/
jitsi-videobridge/lib/jetty-rewrite-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-security-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-serve
r-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-servlet-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-servlets-9.2.10.v20150310.jar:/usr/share
/jitsi-videobridge/lib/jetty-util-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-webapp-7.0.1.v20091125.jar:/usr/share/jitsi-videobridge/lib/jetty-xml-7.0.1.
v20091125.jar:/usr/share/jitsi-videobridge/lib/jicoco-1.1-20180821.201527-7.jar:/usr/share/jitsi-videobridge/lib/jitsi-android-osgi-1.0-20180322.162617-2.jar:/usr/share/j
itsi-videobridge/lib/jitsi-configuration-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-dnsservice-2.9-20150723.181644-1.jar:/usr/share/jitsi-videobridge/lib/jit
si-fileaccess-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-netaddr-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-packetlogging-2.13.f6042d3.jar:/usr/
share/jitsi-videobridge/lib/jitsi-protocol-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-jabber-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi
-protocol-media-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-resourcemanager-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-stats-1.0-20180523.024051-
5.jar:/usr/share/jitsi-videobridge/lib/jitsi-ui-service-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jitsi-util-2.13.f6042d3.jar:/usr/share/jitsi-videobridge/lib/jna
-4.1.0.jar:/usr/share/jitsi-videobridge/lib/jnsapi-0.0.3-jitsi-smack4.2-3.jar:/usr/share/jitsi-videobridge/lib/jose4j-0.5.1.jar:/usr/share/jitsi-videobridge/lib/json-simp
le-1.1.1.jar:/usr/share/jitsi-videobridge/lib/jul-to-slf4j-1.7.7.jar:/usr/share/jitsi-videobridge/lib/jxmpp-core-0.6.0.jar:/usr/share/jitsi-videobridge/lib/jxmpp-jid-0.6.
0.jar:/usr/share/jitsi-videobridge/lib/jxmpp-util-cache-0.6.0.jar:/usr/share/jitsi-videobridge/lib/libidn-1.15.jar:/usr/share/jitsi-videobridge/lib/libjitsi-1.0-20181119.
181042-364.jar:/usr/share/jitsi-videobridge/lib/log4j-1.2.15.jar:/usr/share/jitsi-videobridge/lib/log4j-api-2.3.jar:/usr/share/jitsi-videobridge/lib/log4j-core-2.3.jar:/u
sr/share/jitsi-videobridge/lib/object-cloner-0.1.jar:/usr/share/jitsi-videobridge/lib/objenesis-2.2.jar:/usr/share/jitsi-videobridge/lib/okhttp-3.9.1.jar:/usr/share/jitsi
-videobridge/lib/okio-1.13.0.jar:/usr/share/jitsi-videobridge/lib/orange-extensions-1.3.0.jar:/usr/share/jitsi-videobridge/lib/org.apache.felix.framework-4.4.0.jar:/usr/s
hare/jitsi-videobridge/lib/org.apache.felix.main-4.4.0.jar:/usr/share/jitsi-videobridge/lib/org.osgi.core-4.3.1.jar:/usr/share/jitsi-videobridge/lib/osgi-over-slf4j-1.7.7
.jar:/usr/share/jitsi-videobridge/lib/sdes4j-1.1.3.jar:/usr/share/jitsi-videobridge/lib/sdp-api-1.0.jar:/usr/share/jitsi-videobridge/lib/sigar-1.6.4.jar:/usr/share/jitsi-
videobridge/lib/slf4j-api-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-jdk14-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-simple-1.6.1.jar:/usr/share/jitsi-videob
ridge/lib/smack-bosh-4.2.1.jar:/usr/share/jitsi-videobridge/lib/smack-core-4.2.2-b1c107f.jar:/usr/share/jitsi-videobridge/lib/smack-debug-4.2.1.jar:/usr/share/jitsi-video
root@point:/etc/apache2# netstat -lnp|grep 11798
tcp6       0      0 192.168.2.126:4443      :::*                    LISTEN      11798/java          
udp6       0      0 192.168.2.126:10000     :::*                                11798/java          
udp6       0      0 :::38104                :::*                                11798/java

@damencho
Copy link
Member

Does your deployment have its public address on the local machine, the address that matches the dns you use to access the deployment? If you open 3 tabs on your PC, does it work?

@ghost
Copy link
Author

ghost commented Dec 27, 2018

@damencho Good check, thanks - yep, the public address is on the local machine. While testing locally I'm using localhost as the DNS name currently, and apache2 is listening there on port 443.

When I browse to https://localhost/testing in three separate windows, I'm able to share content in one screen and I see it correctly in the second and third windows.

@ghost
Copy link
Author

ghost commented Dec 27, 2018

Next configuration:

# cat /etc/jitsi/videobridge/sip-communicator.properties 
org.jitsi.videobridge.AUTHORIZED_SOURCE_REGEXP=focus@auth.localhost/.*
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=192.168.2.126
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=192.168.2.126

Results - same behavior:

  • Human participant connects
  • Streamer participant connects
  • Human participant screen turns black, fullscreen in preparation of receiving video
  • Human participant observes that streamer client has 'Connectivity issues'

jvb-natconfig.log

@damencho
Copy link
Member

org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=192.168.2.126
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=192.168.2.126

This is not correct, remove those lines if you are not putting the public address there. From your words, I understand that you test everything in the same network and there is no public address. If this is the case you don't need any setting.
Seems to me that the streamer thing is not able to connect and send media for some reason, maybe the warning/errors you pasted above I don't know. This needs more debugging.

Open chrome://webrtc-internals and find the streamer ssrc/streams and see is anything received at all.

@ghost
Copy link
Author

ghost commented Dec 27, 2018

@damencho I'm not sure exactly what to look for here, but it does appear that some streams are being created in the human participant chrome browser:

chrome-streams

Let me know if I should dump more of this for debugging purposes.

@damencho
Copy link
Member

Check recv ssrc, are there any bytes received

@ghost
Copy link
Author

ghost commented Dec 27, 2018

ssrc-recv

@damencho it appears not - bytesReceived is zero

@ghost
Copy link
Author

ghost commented Dec 29, 2018

@damencho any recommendations on next debugging steps? Glad to provide further details and investigation while this is fresh in my mind. For what it's worth, I do also have a local checkout of ice4j and can apply any relevant patches there for testing purposes (some of the previous log messages led me to believe the problem might be in that area)

@damencho
Copy link
Member

Any errors on the other side, why ice fails there ... Not sure how ti debug the other client you use ...

@ghost
Copy link
Author

ghost commented Dec 29, 2018

@damencho Sounds good; perhaps I'll have to follow this in a chrome-webrtc issue tracker too; I'll leave this issue open until finding out more

@ghost
Copy link
Author

ghost commented Dec 30, 2018

Adding DTLS 1.2 support in libjitsi (by way of recompiling a custom bc-java as per the notes in jitsi/libjitsi#441) seems to help a lot here. That said, there are other issues when trying to use the latest head version of bc-java at the time of writing, so I'm trying to narrow in on any commits which provide a workable system.

@ghost
Copy link
Author

ghost commented Dec 30, 2018

Just for the record:

  • Building against bcgit/bc-java@2ae106c63 has allowed me to get to the point where the webrtc-streamer client connects and no more 'Bunny is having connectivity issues ...' messages appear for the human participant
  • Video still does not appear and I believe this is due to a DTLS handshake failure for the RTP stream - the following error appears in the webrtc-streamer client at high logging verbosity:
[005:363] [16909] (opensslstreamadapter.cc:784): BeginSSL with peer.
[005:363] [16909] (opensslstreamadapter.cc:841): ContinueSSL
[005:364] [16909] (opensslstreamadapter.cc:872):  -- error want read
  • I've tracked the error in the previous item down to an open issue in the webrtc issue tracker: https://bugs.chromium.org/p/webrtc/issues/detail?id=3644 , and have almost managed to rebase and apply one of the proposed patches to fix this, but there's one remaining issue to resolve re: determining whether a server address is 'secure' or not for the purposes of using UDP (insecure) vs DTLS (secure)

@stale
Copy link

stale bot commented Mar 30, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix Issue won't be fixed label Mar 30, 2019
@stale stale bot closed this as completed Apr 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wontfix Issue won't be fixed
Projects
None yet
Development

No branches or pull requests

2 participants