Problem with Cisco 140AC access point.

[IanAber]

I have several devices based on ESP8266 which have been running perfectly until I installed a set of Cisco 140AC access points. Being Cisco I assumed they would be trouble free but they have upset the ESP8266.

The devices all have a small WEB site on them to allow me to configure them for WiFi credentials and to manage the things they control. They also provide data in the form of JSON via a number of api end points. I find the when they are connected to the Cisco 140AC they will happily provide the JSON data but I cannot access the configuration or management pages. I found that it seems to be page size related. I am using the response.sendFile() call to transmit an HTML page to the client and if I keep the size of the page down below approximately 1500 bytes all is well. As soon as I try to send more than that the ESP8266 fails to send anything and the connection times out on the browser requesting it. This does not happen with my older Engenius access points or when I am directly connected to the ESP8266 running in access point mode.

Everything else I have seems to work fine with the Cisco access points, just the ESP8266 is failing.

I tried adding a bunch of debug print output to various library functions and it would seem that after the stream is sent to the client there are a bunch of missing entries when the size of the packet is too big.

This is a sample from a transmission that works...
`
[:135] TCP 3fff2780 timeout updating: 70 -> 2
[:97] Opening connection. Total connections: 2
[:107] TcpServer onClient: 192.168.10.127

[:528] internalOnSent
[:154] onSent
[:155] TCP 3fff2488 sent: 1564
[:158] TCP 3fff2488 4 64
[:205] onReadyToSendData
[:207] State : eHCS_Ready
[:168] TCPClient::readyToSendData(2)
[:212] onReadyToSendData
[:214] TCP 3fff2488 onReadyToSendData: 2
[:174] No delegate sourceEvent:ready
[:196] Stream is null
[:532] TCP 3fff2488 <sent
[:538] staticOnPoll
[:551] internalOnPoll
[:167] TCP 3fff2488 TcpConnection::onPoll() sleep = 1 : timeOut = 2
[:174] TCP 3fff2488 1 - 2
[:205] onReadyToSendData
[:207] State : eHCS_Ready
[:168] TCPClient::readyToSendData(3)
[:212] onReadyToSendData
[:174] No delegate sourceEvent:ready
`

Tihs is the same section from where the transmission fails...

`
:135] TCP 3fff2928 timeout updating: 70 -> 2
[:97] Opening connection. Total connections: 2
[:107] TcpServer onClient: 192.168.10.127

[:538] staticOnPoll
[:551] internalOnPoll
[:167] TCP 3fff2508 TcpConnection::onPoll() sleep = 1 : timeOut = 2
[:174] TCP 3fff2508 1 - 2
[:205] onReadyToSendData
[:207] State : eHCS_Ready
[:168] TCPClient::readyToSendData(3)
[:212] onReadyToSendData
[:174] No delegate sourceEvent:ready
`

The file names are missing from the debug output due to another bug which I logged the other day but the line numbers are useful.
The ESP8266 remains responsive and does not seem at all phased by the failure to transmit the packet. If I try to send a string immediately after the file the string is sent to the client correctly but there is no file.

[slaff]

This might be due to the TCP Maximum segment size (TCP_MSS). You can try to change it and test. What you can do is edit
the following file $SMING_HOME/Arch/Esp8266/Components/esp-open-lwip/esp-open-lwip/include/lwipopts.h and change the value of #define TCP_MSS 1390 to something higher or lower. Once ready with the changes in the file you have to recompile the lwip package and relink your application. So every time you change the TCP_MSS value you should.

0. Change the current directory to the root folder of your application.
1. Edit the $SMING_HOME/Arch/Esp8266/Components/esp-open-lwip/esp-open-lwip/include/lwipopts.h and change the TCP_MSS value.
2. Recompile lwip using the following command:
   make esp-open-lwip-rebuild
3. Relink your application using the following command:
   make clean && make
4. Flash the application on the device.
   make flashapp

[IanAber]

Many thanks for the speedy response. You guys are top notch!

I looked at the header file and the comment told me that the default should have been set to 536 but it was actually set to 1390. I did not make that change so I assume it had been done in the distribution some time ago. I changed it to 536 and the system seems to be working now. I have not updated all of the devices because one has been installed for several years and the code I have will not recompile on the latest version of the development system using VSCode. This, perhaps, shows how long this change might have been made, at least 3 or 4 years ago I am guessing.

Am I correct in thinking that the actual size is calculated based on the information from the receiver when the connection is first made? Could it be that the Cisco access point is providing a capability of handling a larger buffer and somehow this is not being calculated correctly or the ESP8266 is running out of memory?

How does this mechanism work? I would love to be able to help troubleshoot this since it appears that I have an environment in which I can reliably make it fail. How can I see what is coming from the access point?

[slaff]

I guess the LWIP stack and Cisco cannot correctly negotiate a MSS value. My blind guess is that Cisco is deeming 1390 as too high or not matching some criteria and suggesting another value. And may be here the LWIP stack is not flexible enough to use the suggested value.
You can run Wireshark on your laptop, sniff you WIFI connection (using promiscuous mode) and then see what is happening with the TCP SYN packets. Wireshark will show you the MSS value sent from the initiator of the connection - the ESP8266 device and then you will be able to see the response MSS from Cisco.
Or you can turn LWIP debugging and add some debug messages in $SMING_HOME/Arch/Esp8266/Components/esp-open-lwip/esp-open-lwip/lwip/core/tcp.c around line 675 where the tcp_connect function is defined.

More about MSS from Wikipedia:
https://en.wikipedia.org/wiki/Maximum_segment_size

[slaff]

@IanAber this might also be an interesting read: https://tailscale.com/blog/throughput-improvements/. Especially the part about "TCP Segmentation"