Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OS X: Nothing received from server on UDP port 60003 #1254

Open
jaidetree opened this issue Jan 29, 2023 · 37 comments
Open

OS X: Nothing received from server on UDP port 60003 #1254

jaidetree opened this issue Jan 29, 2023 · 37 comments

Comments

@jaidetree
Copy link

jaidetree commented Jan 29, 2023

Reproduction

  1. Ran mosh-server and got the following output
> mosh-server

MOSH CONNECT 60003 SaHaHCFHl3b93sZkEsZDZA

mosh-server (mosh 1.4.0) [build mosh-1.3.2.95rc1-13-gc516fb4-dirty]
Copyright 2012 Keith Winstein <[email protected]>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 83878]
  1. Copied the key
  2. Ran env MOSH_KEY="SaHaHCFHl3b93sZkEsZDZA" mosh-client 192.168.0.200 60003

Expected

The client should connect to the mosh-server and display my login shell

Actual

Instead it seems to make initial ssh connection, but the mosh connection hangs:

mosh: Nothing received from server on UDP port 60003

image

Environment

Key Value
Operating System OS X Ventura 13.1
Mosh version mosh 1.4.0 [build mosh-1.3.2.95rc1-13-gc516fb4-dirty]
Local Network IP 192.168.0.200

What I've tried

  • Followed the debug guide on the wiki and used the netcat command to listen on UDP port 60003, then used netcat in another terminal pane to send a message. It worked as expected.
  • It does work on localhost env MOSH_KEY="<key>" mosh-client 127.0.0.1 60003 but not my local network IP like the example above
  • Disabled the firewall on OS X and my router, setup port forwarding as well

Any help would be appreciated!

@achernya
Copy link
Collaborator

Please re-try using the mosh wrapper client. mosh-client and mosh-server are not intended to be invoked directly. My suspicion is you were not fast enough and the timeout on mosh-server to exit if it hadn't heard from a client within the first 60 seconds triggered.

Please re-open if you continue to have issues, and also please if macOS is the client or the server or both.

@jaidetree
Copy link
Author

jaidetree commented Jan 29, 2023

macOS is the client and server, just trying to get a connection over my LAN ip on the same machine to work. It's not a speed issue, I'm able to do it with localhost and my LAN ip 192.168.0.200 all within 1 minute. I'm happy to record it to prove that's not the case.

Tried with the mosh wrapper, but seem to get the same issue:

Localhost works.

> mosh localhost --server=$(which mosh-server)

Using my current local ip does not, which should point to the same machine:

> mosh 192.168.0.200 --server=$(which mosh-server)

Gives me the same error as above.

Lastly, I cannot reopen this ticket, I don't have any UI options to do that.

@achernya achernya reopened this Jan 29, 2023
@achernya achernya added the OS X label Jan 29, 2023
@achernya
Copy link
Collaborator

achernya commented Jan 29, 2023

It sounds like you have an issue with your firewall if a specific IP does not work. I also tried to reproduce this with a default-config Mac (Ventura 13.2) and could not. I did have to enable remote login in the Sharing pane.

Have you confirmed that 192.168.0.200 is the correct IP? Can you paste the output of ifconfig?

@jaidetree
Copy link
Author

jaidetree commented Jan 30, 2023

  • Disabled the firewall on OS X and my router, setup port forwarding as well

Already disabled it, as that seemed like a common issue.

  • Followed the debug guide on the wiki and used the netcat command to listen on UDP port 60003, then used netcat in another terminal pane to send a message. It worked as expected.

Again, used netcat to listen on UDP 60003 and was able to receive messages across 192.168.0.200 from the same machine, and other devices on the network. So doesn't appear to be a firewall issue.

ifconfig:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
	inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
	nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en6: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether ac:de:48:00:11:22 
	inet6 fe80::aede:48ff:fe00:1122%en6 prefixlen 64 scopeid 0x4 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect (100baseTX <full-duplex>)
	status: active
ap1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=6463<RXCSUM,TXCSUM,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
	ether 36:7d:da:7e:58:68 
	inet6 fe80::347d:daff:fe7e:5868%ap1 prefixlen 64 scopeid 0x6 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether 14:7d:da:7e:58:68 
	inet6 fe80::c95:b66d:3cfe:175%en0 prefixlen 64 secured scopeid 0x7 
	inet 192.168.0.169 netmask 0xffffff00 broadcast 192.168.0.255
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
en4: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=460<TSO4,TSO6,CHANNEL_IO>
	ether 82:2d:0e:81:18:04 
	media: autoselect <full-duplex>
	status: inactive
en3: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=460<TSO4,TSO6,CHANNEL_IO>
	ether 82:2d:0e:81:18:05 
	media: autoselect <full-duplex>
	status: inactive
en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=460<TSO4,TSO6,CHANNEL_IO>
	ether 82:2d:0e:81:18:01 
	media: autoselect <full-duplex>
	status: inactive
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=460<TSO4,TSO6,CHANNEL_IO>
	ether 82:2d:0e:81:18:00 
	media: autoselect <full-duplex>
	status: inactive
en7: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=6467<RXCSUM,TXCSUM,VLAN_MTU,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
	ether 3c:18:a0:43:a0:43 
	inet6 fe80::85b:ad2d:d944:58fb%en7 prefixlen 64 secured scopeid 0xc 
	inet 192.168.0.200 netmask 0xffffff00 broadcast 192.168.0.255
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect (1000baseT <full-duplex>)
	status: active
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=63<RXCSUM,TXCSUM,TSO4,TSO6>
	ether 82:2d:0e:81:18:01 
	Configuration:
		id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
		maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
		root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
		ipfilter disabled flags 0x0
	member: en1 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 10 priority 0 path cost 0
	member: en2 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 11 priority 0 path cost 0
	member: en3 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 9 priority 0 path cost 0
	member: en4 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 8 priority 0 path cost 0
	nd6 options=201<PERFORMNUD,DAD>
	media: <unknown type>
	status: inactive
awdl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=6463<RXCSUM,TXCSUM,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
	ether c6:33:c6:c1:fd:16 
	inet6 fe80::c433:c6ff:fec1:fd16%awdl0 prefixlen 64 scopeid 0xe 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
llw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether c6:33:c6:c1:fd:16 
	inet6 fe80::c433:c6ff:fec1:fd16%llw0 prefixlen 64 scopeid 0xf 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: inactive
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::a807:4bd4:da5d:e9d9%utun0 prefixlen 64 scopeid 0x10 
	nd6 options=201<PERFORMNUD,DAD>
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
	inet6 fe80::76cf:1146:8293:e52d%utun1 prefixlen 64 scopeid 0x11 
	nd6 options=201<PERFORMNUD,DAD>
utun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1000
	inet6 fe80::ce81:b1c:bd2c:69e%utun2 prefixlen 64 scopeid 0x12 
	nd6 options=201<PERFORMNUD,DAD>
utun3: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::5f28:5a28:b46a:1b50%utun3 prefixlen 64 scopeid 0x16 
	nd6 options=201<PERFORMNUD,DAD>
utun4: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::9d05:6192:1124:2d22%utun4 prefixlen 64 scopeid 0x17 
	nd6 options=201<PERFORMNUD,DAD>
utun5: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::432f:86de:71d:55e%utun5 prefixlen 64 scopeid 0x18 
	nd6 options=201<PERFORMNUD,DAD>
utun6: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::19dd:7b9d:9bc5:55a0%utun6 prefixlen 64 scopeid 0x19 
	nd6 options=201<PERFORMNUD,DAD>
utun7: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::768f:679b:21c8:79c7%utun7 prefixlen 64 scopeid 0x1b 
	nd6 options=201<PERFORMNUD,DAD>
utun8: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::27b9:36cf:722d:b729%utun8 prefixlen 64 scopeid 0x1c 
	nd6 options=201<PERFORMNUD,DAD>
utun9: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::1c86:657d:cccb:9d53%utun9 prefixlen 64 scopeid 0x1d 
	nd6 options=201<PERFORMNUD,DAD>
utun10: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::8a78:de50:5912:897d%utun10 prefixlen 64 scopeid 0x1e 
	nd6 options=201<PERFORMNUD,DAD>
en9: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether 6e:4a:85:a8:df:21 
	inet6 fe80::18c5:49ea:a9b4:629b%en9 prefixlen 64 secured scopeid 0x1a 
	inet 169.254.224.61 netmask 0xffff0000 broadcast 169.254.255.255
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect (100baseTX <full-duplex>)
	status: active
en10: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether d6:ad:33:92:40:23 
	inet6 fe80::d4ad:33ff:fe92:4023%en10 prefixlen 64 scopeid 0x5 
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active

@achernya
Copy link
Collaborator

It looks like you have 2 connections active in the same subnet -- one with IP address 192.168.0.169 and the other with 192.168.0.200. Can you try disabling the .169 one and re-trying?

@jaidetree
Copy link
Author

Disabled wifi, the 192.168.0.169 interface, and ran:

mosh 192.168.0.200 --server=$(which mosh-server)

Same issue unfortunately

@jaidetree
Copy link
Author

image

Did the netcat test again and was able to receive the message

@achernya
Copy link
Collaborator

Can you run mosh-server and immediately after it,lsof -i -n | grep mosh and paste the results of both? I want to make sure mosh-server is actually running and listening.

You should see something like

$ mosh-server  
MOSH CONNECT 60001 cwNZTxaCw8N1ZgGNdkzYjg

mosh-server (mosh 1.4.0) [build mosh 1.4.0]
Copyright 2012 Keith Winstein <[email protected]>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 5763]
$ lsof -i -n | grep mosh                             
mosh-serv 5763 achernya    5u  IPv6 0x8f707611276235a9      0t0  UDP *:60001

Also, out of curiosity, do you have any VMs or VPNs running on this machine? I'm only asking because both of those things could potentially mess with routing tables in weird ways, but that is admittedly inconsistent with nc working.

Also, I will quickly mention that the error message you're getting from mosh-client is that it's not heard anything back from the mosh server, while your nc tests show the server can hear from the client. So you really should try typing in the server window, not the client window, if you want to reproduce the situation mosh is encountering.

@achernya
Copy link
Collaborator

Also, another silly question: do you have anything in your ~/.ssh/config?

@jaidetree
Copy link
Author

mosh-server and lsof

> mosh-server


MOSH CONNECT 60004 GqY0VoZfWHcyvdOhRgyeyA

mosh-server (mosh 1.4.0) [build mosh-1.3.2.95rc1-13-gc516fb4-dirty]
Copyright 2012 Keith Winstein <[email protected]>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 6032]
j at j-crunchtop-mbp in ~/p/clj-piano on develop with new & changed files
 > lsof -i -n | grep mosh
mosh-serv  6032    j    5u  IPv6 0x9e56ee9f6a680a13      0t0  UDP *:60004

Current ssh config

 > cat ~/.ssh/config
Host *
   AddKeysToAgent yes
   UseKeychain yes
   ServerAliveInterval 10
   ControlPath ~/.ssh/controlmasters/%r@%h:%p
   ControlMaster auto
   ControlPersist yes
   IdentityFile ~/.ssh/id_rsa
   IdentityFile ~/.ssh/id_rsa_kc
   IdentityFile ~/.ssh/id_ed25519

@achernya
Copy link
Collaborator

I believe ControlMaster is causing you problems. Can you try mosh --ssh="ssh -F none" --server=$(which mosh-server) 192.168.0.200? If I'm correct, then the mosh option --experimental-remote-ip=remote would be a solution to your problem.

@jaidetree
Copy link
Author

  • Used mosh --sh="ssh -F none" --server=$(which mosh-server) but did not work
  • --exerimental-remote-ip=remote flag also did not work
  • Removed those Control* options from my ssh config, don't recall why I added them

@achernya
Copy link
Collaborator

Did you perhaps make a typo? it's --ssh not --sh. I am however at a bit of a loss at this point. Can we try starting a packet capture (sudo tcpdump -i all 'udp and portrange 60000-60010' -n -v) before running mosh and seeing what happens?

@jaidetree
Copy link
Author

Sorry the typo was with my reply. Pasted it from your reply to run it. Thanks for trying at least, I'm stumped as well.

2023-01-29 20 17 29

Capture output:

tcpdump: data link type PKTAP
tcpdump: listening on all, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes
20:17:35.547968 IP (tos 0x2,ECT(0), ttl 64, id 46687, offset 0, flags [none], proto UDP (17), length 106, bad cksum 0 (->4141)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 78
20:17:35.547980 IP (tos 0x2,ECT(0), ttl 64, id 46687, offset 0, flags [none], proto UDP (17), length 106, bad cksum 0 (->4141)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 78
20:17:38.549142 IP (tos 0x2,ECT(0), ttl 64, id 36487, offset 0, flags [none], proto UDP (17), length 103, bad cksum 0 (->691c)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 75
20:17:38.549184 IP (tos 0x2,ECT(0), ttl 64, id 36487, offset 0, flags [none], proto UDP (17), length 103, bad cksum 0 (->691c)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 75
20:17:40.070441 IP (tos 0x2,ECT(0), ttl 64, id 22082, offset 0, flags [none], proto UDP (17), length 103, bad cksum 0 (->a161)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 75
20:17:40.070450 IP (tos 0x2,ECT(0), ttl 64, id 22082, offset 0, flags [none], proto UDP (17), length 103, bad cksum 0 (->a161)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 75
20:17:40.321775 IP (tos 0x2,ECT(0), ttl 64, id 36925, offset 0, flags [none], proto UDP (17), length 103, bad cksum 0 (->6766)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 75
20:17:40.321787 IP (tos 0x2,ECT(0), ttl 64, id 36925, offset 0, flags [none], proto UDP (17), length 103, bad cksum 0 (->6766)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 75
20:17:40.576182 IP (tos 0x2,ECT(0), ttl 64, id 13764, offset 0, flags [none], proto UDP (17), length 112, bad cksum 0 (->c1d6)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 84
20:17:40.576195 IP (tos 0x2,ECT(0), ttl 64, id 13764, offset 0, flags [none], proto UDP (17), length 112, bad cksum 0 (->c1d6)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 84
20:17:40.826375 IP (tos 0x2,ECT(0), ttl 64, id 22984, offset 0, flags [none], proto UDP (17), length 110, bad cksum 0 (->9dd4)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 82
20:17:40.826416 IP (tos 0x2,ECT(0), ttl 64, id 22984, offset 0, flags [none], proto UDP (17), length 110, bad cksum 0 (->9dd4)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 82
20:17:41.077328 IP (tos 0x2,ECT(0), ttl 64, id 46607, offset 0, flags [none], proto UDP (17), length 109, bad cksum 0 (->418e)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 81
20:17:41.077342 IP (tos 0x2,ECT(0), ttl 64, id 46607, offset 0, flags [none], proto UDP (17), length 109, bad cksum 0 (->418e)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 81
20:17:41.330551 IP (tos 0x2,ECT(0), ttl 64, id 44764, offset 0, flags [none], proto UDP (17), length 103, bad cksum 0 (->48c7)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 75
20:17:41.330563 IP (tos 0x2,ECT(0), ttl 64, id 44764, offset 0, flags [none], proto UDP (17), length 103, bad cksum 0 (->48c7)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 75
20:17:41.583181 IP (tos 0x2,ECT(0), ttl 64, id 25382, offset 0, flags [none], proto UDP (17), length 101, bad cksum 0 (->947f)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 73
20:17:41.583205 IP (tos 0x2,ECT(0), ttl 64, id 25382, offset 0, flags [none], proto UDP (17), length 101, bad cksum 0 (->947f)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 73
20:17:41.837830 IP (tos 0x2,ECT(0), ttl 64, id 45751, offset 0, flags [none], proto UDP (17), length 104, bad cksum 0 (->44eb)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 76
20:17:41.837851 IP (tos 0x2,ECT(0), ttl 64, id 45751, offset 0, flags [none], proto UDP (17), length 104, bad cksum 0 (->44eb)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 76
20:17:42.092978 IP (tos 0x2,ECT(0), ttl 64, id 40155, offset 0, flags [none], proto UDP (17), length 102, bad cksum 0 (->5ac9)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 74
20:17:42.092989 IP (tos 0x2,ECT(0), ttl 64, id 40155, offset 0, flags [none], proto UDP (17), length 102, bad cksum 0 (->5ac9)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 74
20:17:42.347634 IP (tos 0x2,ECT(0), ttl 64, id 36180, offset 0, flags [none], proto UDP (17), length 99, bad cksum 0 (->6a53)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 71
20:17:42.347673 IP (tos 0x2,ECT(0), ttl 64, id 36180, offset 0, flags [none], proto UDP (17), length 99, bad cksum 0 (->6a53)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 71
20:17:42.601999 IP (tos 0x2,ECT(0), ttl 64, id 28496, offset 0, flags [none], proto UDP (17), length 109, bad cksum 0 (->884d)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 81
20:17:42.602021 IP (tos 0x2,ECT(0), ttl 64, id 28496, offset 0, flags [none], proto UDP (17), length 109, bad cksum 0 (->884d)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 81
20:17:42.852436 IP (tos 0x2,ECT(0), ttl 64, id 3267, offset 0, flags [none], proto UDP (17), length 111, bad cksum 0 (->ead8)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 83
20:17:42.852454 IP (tos 0x2,ECT(0), ttl 64, id 3267, offset 0, flags [none], proto UDP (17), length 111, bad cksum 0 (->ead8)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 83
20:17:43.104821 IP (tos 0x2,ECT(0), ttl 64, id 41944, offset 0, flags [none], proto UDP (17), length 101, bad cksum 0 (->53cd)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 73
20:17:43.104862 IP (tos 0x2,ECT(0), ttl 64, id 41944, offset 0, flags [none], proto UDP (17), length 101, bad cksum 0 (->53cd)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 73
20:17:43.358272 IP (tos 0x2,ECT(0), ttl 64, id 60445, offset 0, flags [none], proto UDP (17), length 106, bad cksum 0 (->b83)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 78
20:17:43.358296 IP (tos 0x2,ECT(0), ttl 64, id 60445, offset 0, flags [none], proto UDP (17), length 106, bad cksum 0 (->b83)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 78
20:17:43.613038 IP (tos 0x2,ECT(0), ttl 64, id 6737, offset 0, flags [none], proto UDP (17), length 99, bad cksum 0 (->dd56)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 71
20:17:43.613085 IP (tos 0x2,ECT(0), ttl 64, id 6737, offset 0, flags [none], proto UDP (17), length 99, bad cksum 0 (->dd56)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 71
20:17:43.866844 IP (tos 0x2,ECT(0), ttl 64, id 49593, offset 0, flags [none], proto UDP (17), length 115, bad cksum 0 (->35de)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 87
20:17:43.866862 IP (tos 0x2,ECT(0), ttl 64, id 49593, offset 0, flags [none], proto UDP (17), length 115, bad cksum 0 (->35de)!)
    192.168.0.200.56502 > 192.168.0.200.60001: UDP, length 87

@achernya
Copy link
Collaborator

From the packet capture, we can see that the mosh-client is in fact trying to contact the mosh-server, but it never replies. If this were linux, I'd say to strace mosh-server and see what it's doing, but unfortunately, this is macOS, and I don't know dtrace/dtruss well enough to form a useful suggestion.

It really does seem like there's something weird with your network. It feels like a firewall dropping the packets. The packets are small, so it isn't something like an MTU issue (not that that should be a problem locally).

This is what it should the tcpdump should look like:

tcpdump: listening on all, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes
20:47:12.645998 IP (tos 0x2,ECT(0), ttl 64, id 29101, offset 0, flags [none], proto UDP (17), length 155)
    192.168.1.87.60001 > 192.168.1.48.49466: UDP, length 127
20:47:12.684764 IP (tos 0x2,ECT(0), ttl 64, id 13442, offset 0, flags [none], proto UDP (17), length 93, bad cksum 0 (->c25b)!)
    192.168.1.48.60001 > 192.168.1.48.60664: UDP, length 65
20:47:12.684791 IP (tos 0x2,ECT(0), ttl 64, id 13442, offset 0, flags [none], proto UDP (17), length 93, bad cksum 0 (->c25b)!)
    192.168.1.48.60001 > 192.168.1.48.60664: UDP, length 65
20:47:12.751578 IP (tos 0x2,ECT(0), ttl 64, id 61054, offset 0, flags [none], proto UDP (17), length 105)
    192.168.1.48.49466 > 192.168.1.87.60001: UDP, length 77
20:47:12.832234 IP (tos 0x2,ECT(0), ttl 64, id 1872, offset 0, flags [none], proto UDP (17), length 96, bad cksum 0 (->ef8a)!)
    192.168.1.48.60664 > 192.168.1.48.60001: UDP, length 68
20:47:12.832272 IP (tos 0x2,ECT(0), ttl 64, id 1872, offset 0, flags [none], proto UDP (17), length 96, bad cksum 0 (->ef8a)!)
    192.168.1.48.60664 > 192.168.1.48.60001: UDP, length 68

so just in case you were wonder if the bad cksum is a problem, it's not, that's just a sign of checksum offloading to the NIC.

Just to confirm, while you were doing this, there was a mosh-server running? And it was on the port that you see in the pcap? The lsof commands from before should help you confirm this. Another experiment to do is to get a packet capture of your nc tests and see how they compare.

@jaidetree
Copy link
Author

Tried both running mosh --ssh="ssh -F none" --server=$(which mosh-server) and the manual mosh-server and mosh-client methods in substantially < 1 minute.

mosh-server localhost --server=$(which mosh-server) does work, same with mosh-server and mosh-client when using to 127.0.0.1.

It's possible it's a networking issue, but why does netcat work and receive messages correctly?

@jaidetree
Copy link
Author

2023-01-29 20 59 37

 > sudo tcpdump -i all 'udp and portrange 60000-60010' -n -v
tcpdump: data link type PKTAP
tcpdump: listening on all, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes
20:59:39.211075 IP (tos 0x0, ttl 64, id 34541, offset 0, flags [none], proto UDP (17), length 29, bad cksum 0 (->7102)!)
    192.168.0.200.49274 > 192.168.0.200.60003: UDP, length 1
20:59:39.211086 IP (tos 0x0, ttl 64, id 59918, offset 0, flags [none], proto UDP (17), length 29, bad cksum 0 (->de1)!)
    192.168.0.200.49274 > 192.168.0.200.60003: UDP, length 1
20:59:39.211087 IP (tos 0x0, ttl 64, id 34541, offset 0, flags [none], proto UDP (17), length 29, bad cksum 0 (->7102)!)
    192.168.0.200.49274 > 192.168.0.200.60003: UDP, length 1
20:59:39.211091 IP (tos 0x0, ttl 64, id 29590, offset 0, flags [none], proto UDP (17), length 29, bad cksum 0 (->8459)!)
    192.168.0.200.49274 > 192.168.0.200.60003: UDP, length 1
20:59:39.211095 IP (tos 0x0, ttl 64, id 32349, offset 0, flags [none], proto UDP (17), length 29, bad cksum 0 (->7992)!)
    192.168.0.200.49274 > 192.168.0.200.60003: UDP, length 1
20:59:39.211171 IP (tos 0x0, ttl 64, id 59918, offset 0, flags [none], proto UDP (17), length 29, bad cksum 0 (->de1)!)
    192.168.0.200.49274 > 192.168.0.200.60003: UDP, length 1
20:59:39.211172 IP (tos 0x0, ttl 64, id 29590, offset 0, flags [none], proto UDP (17), length 29, bad cksum 0 (->8459)!)
    192.168.0.200.49274 > 192.168.0.200.60003: UDP, length 1
20:59:39.211173 IP (tos 0x0, ttl 64, id 32349, offset 0, flags [none], proto UDP (17), length 29, bad cksum 0 (->7992)!)
    192.168.0.200.49274 > 192.168.0.200.60003: UDP, length 1
20:59:39.219722 IP (tos 0x0, ttl 64, id 38030, offset 0, flags [none], proto UDP (17), length 40, bad cksum 0 (->6356)!)
    192.168.0.200.49274 > 192.168.0.200.60003: UDP, length 12
20:59:39.219738 IP (tos 0x0, ttl 64, id 38030, offset 0, flags [none], proto UDP (17), length 40, bad cksum 0 (->6356)!)
    192.168.0.200.49274 > 192.168.0.200.60003: UDP, length 12

That was the dump from running the nc tests again.

@achernya
Copy link
Collaborator

Without a tcpdump to compare, it's hard to say. But as I mentioned before, your netcat is testing the opposite datastream, we don't actually know for certain it works.

localhost working doesn't necessarily mean a firewall issue is not happening, as firewalls can and do consider input/output interfaces.

@jaidetree
Copy link
Author

jaidetree commented Jan 30, 2023

Edit: Netcat test with tcpdump results are above

Archer A7 Router:
image

System Settings:
image

Also tried echo "hello world" | nc -4 -v -u 192.168.0.200 60003 from a tablet, and it also made it to the udp 60003 nc server on my computer. Not feeling confident it's a networking issue directly. Maybe I should try starting the nc server through ssh, then try connecting to that?

@jaidetree
Copy link
Author

Tried running the nc server from an ssh connection, created a new tab in blink shell and wrote to the 192.168.0.200 60003 port and that also worked, at least in terms of seeing that message.

Definitely at a loss on this one

@achernya
Copy link
Collaborator

The router on your firewall is not related. The macOS firewall could be, especially since you have a VPN installed. I know it says inactive, but something is definitely weird here.

Can you try connecting to your MBP from a different device using mosh?

@jaidetree
Copy link
Author

That was how I started, I was trying to use mosh to connect from a tablet to my mbp using blink shell but I kept getting that same issue, that's when I started pursuing trying to connect to the mbp from itself to test\debug.

@nburnett
Copy link

Any progress on this? I’m having the same issue trying to connect from Blink On an iPad -> MacOs Ventura

@achernya
Copy link
Collaborator

The developers have no way to reproduce this. I cannot reproduce this on my Macbook Pro, and none of us have an iPad. So unfortunately progress here is unlikely to happen unless someone from the community figures it out.

@nburnett
Copy link

Thanks for the prompt reply! If I ever figure it out I'll post something.

@lloeki
Copy link

lloeki commented Apr 28, 2023

Reporting the same issue here (on two different machines), although at least nc "agrees" and fails to communicate over anything else than localhost.

@carloscabanero
Copy link

@nburnett feel free to reach out to us on Blink issues.

Not sure what would be going on here, but most of our users just need to add rules to their firewall. Leaving it here in case it helps (via rrgeorge):

I have to run this every time i reboot:

#!/usr/bin/env bash
  FW=/usr/libexec/ApplicationFirewall/socketfilterfw
  MOSH=$(/opt/homebrew/bin/brew info mosh|grep Cellar|awk '{print $1}')/bin/mosh  -server
  sudo $FW --add $MOSH
  sudo $FW --unblockapp $MOSH

@nreilly
Copy link

nreilly commented Apr 20, 2024

I think I might be experiencing the same issue currently. One thing I would like to point out, is in #1254 (comment) lsof indicates mosh-server is only bound to UDP via IPv6, while the client is connecting via IPv4. Should it not be listening on both IPv4 and IPv6?

I seem to be experiencing the same challenge when connecting to the IPv4 address on the interface, but it connects fine when I use the IPv6 (automatically assigned) address.

@jaidetree
Copy link
Author

jaidetree commented Apr 20, 2024

Whoa that might be it! Great find

@jaidetree
Copy link
Author

jaidetree commented Apr 20, 2024

Took a few years but finally stumbled on a solution thanks to rrgeorge in the blink shell discord:

#!/usr/bin/env bash

FW=/usr/libexec/ApplicationFirewall/socketfilterfw
MOSH=$(sudo -iu '#501' /usr/local/bin/brew info mosh|grep Cellar|awk '{print $1}')/bin/mosh-server
echo $MOSH
sudo $FW --add $MOSH
sudo $FW --unblockapp $MOSH

For some reason using the firewall cli makes a big difference compared to the Mac OS GUI Firewall settings to add the mosh-server binary.

Unfortunately, I don't know why it works or what the difference is. There's also a few issues in blinkshell with identical solutions, just never found those before.

@bulkan
Copy link

bulkan commented Oct 8, 2024

This problem started happening to me after upgrading to Sequoia on my M2 Mini running mosh-server 1.4.0. I never had the firewall enabled. I can ssh but mosh won't work from the Blink app nor another OSX machine.

Running lsof -i -n | grep mosh I get;

mosh-serv 87822 bulkan    4u  IPv4  0xac6a2e648cfcddf      0t0    UDP 192.168.1.29:60001

Enabling the firewall and using the multiple scripts out there to add mosh-server to allow incoming connections doesn't work either.

@carloscabanero
Copy link

This seems to be an issue in Sequoia with mosh-server. It seems like UDP traffic is having issues in Sequoia, and apparently some VPNs, etc... have also been impacted by it. Firewalls are not related. And I can do a nc client-server on those ports just fine. I'm investigating on the Blink side to see what is going on and playing with the mosh-sever a bit (blinksh/blink#2054). Any pointers are welcome.

@carloscabanero
Copy link

carloscabanero commented Oct 8, 2024

I did a few other quick tests. So when running client-server, the client is not receiving back from mosh-server, but the server is receiving from the client. I traced a running mosh-server and started typing on the "non-receiving client":

 ~ % sudo dtruss -p 940 -t write
SYSCALL(args) 		 = return
write(0x7, "j\0", 0x1)		 = 1 0
write(0x7, "h\0", 0x1)		 = 1 0
write(0x7, "fg\0", 0x2)		 = 2 0
write(0x7, "j\0", 0x1)		 = 1 0
write(0x7, "g\0", 0x1)		 = 1 0
write(0x7, "l\0", 0x1)		 = 1 0
write(0x7, "k\0", 0x1)		 = 1 0
...
write(0x7, "\r\0", 0x1)		 = 1 0
write(0x7, "l\006\0", 0x1)		 = 1 0
write(0x7, "kj\0", 0x2)		 = 2 0
write(0x7, "lkj\0", 0x3)		 = 3 0
write(0x7, "lkj\0", 0x3)		 = 3 0

And from tcpdump, only traffic from Client to Server. I can see the heartbeats and keystrokes just going one way.

~ % sudo tcpdump port 60001
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pktap, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes
15:57:46.740094 IP ipad.50061 > macbookair.60001: UDP, length 74
15:57:49.708756 IP ipad.50061 > macbookair.60001: UDP, length 77
15:57:52.780661 IP ipad.62441 > macbookair.60001: UDP, length 64
15:57:54.727691 IP ipad.62441 > macbookair.60001: UDP, length 83
15:57:54.727697 IP ipad.62441 > macbookair.60001: UDP, length 74

Workaround: If connecting through Tailscale, things still work perfectly. This still looks like some firewall issue on Sequoia.

@carloscabanero
Copy link

We have found the issue in macOS Sequoia (thanks to @rrgeorge). The problem is due to the new Local Network Privacy feature, which was previously only on iOS but is now on macOS as well. Now, whenever apps access the local network, they prompt the user for permission, which is good for regular apps. However, since we're running mosh-server over an SSH connection, that prompt does not appear. We tested this in two ways:

  1. Running mosh-server with sudo (root) bypasses this limitation, and everything works as expected.
  2. Terminal.app is exempt at the system level, so running mosh-server and connecting the client manually also works.
  3. This aligns with earlier tests where the client could reach the server, but not vice versa. Tools like Tailscale may also work since they use different network paths.

To make this new privacy feature compatible with mosh-server, we face the challenge
that at the moment there are no APIs to control it. We need a way to trigger a server connection to the local network running mosh-server on its own. We cannot do it from the Terminal, as it is exempted. We managed to trigger it within a launchd daemon, but the prompt appeared for a "bash" wrapper instead of mosh-server. Running mosh-server over SSH may present its own issues too, as SSH is treated as a separate session and won’t prompt the user.

I don’t have any solutions right now, but any other ideas to bend this backwards and somehow trigger a prompt for mosh-server are welcome. This will affect macOS Sequoia users and force them to use Tailscale or something similar. I will reach out on the Apple developer forums to see if there’s anything I might have missed for CLI apps.

@nreilly
Copy link

nreilly commented Oct 10, 2024

I don’t have any solutions right now, but any other ideas to bend this backwards and somehow trigger a prompt for mosh-server are welcome. This will affect macOS Sequoia users and force them to use Tailscale or something similar. I will reach out on the Apple developer forums to see if there’s anything I might have missed for CLI apps.

I found this a comment on a thread from 2020 that may help.

@rrgeorge
Copy link

In my experiments, listening services only get triggered on a connection. I have a theory that if mosh-server just attempts to make a connection to something on the local network (maybe back to SSH_CLIENT) then it will trigger the prompt.
So my thought is that maybe then your normal initial connection could be calling mosh-server with a special switch (like --local-net-authorize?) that will trigger an outbound connection to trigger the dialog will get triggered. You would only need to do this once (or periodically if that permission is cleared somehow)

@mjhsieh
Copy link

mjhsieh commented Nov 5, 2024

I am getting this error after upgrading to Sequoia 15.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants