I'm writing a semi-realtime application for an embedded board running BusyBox which packages and sends some telemetry data (around 100KB) on a TCP socket every 100 milliseconds. This board uses the RTL8723BS wifi module as its network interface. However on the receiving side, it seems like the data received is 300ms behind what it should be. I've used other embedded boards as well as more powerful computers on the receiving side and the latency has always been around 300ms, so I'm pretty sure it's the sender's fault.

By doing some diagnosis of my own and by changing and observing the sockopts, I found out that the reason for the 300ms latency is because the kernel's TCP write buffer is filled to the maximum! By trial and error, I found that

echo 4096 290000 290000 > /proc/sys/net/ipv4/tcp_wmem

offers the best latency, decreasing the TCP write buffer size below 290,000 will result in dropped packets and increasing it will slightly increase the latency. I tried messing with the sockopts of the sender socket, like setting the TCP_NODELAY option but I couldn't decrease the network latency.

Any ideas why the kernel keeps the data I send() in its TCP buffer instead of immediately sending it out on the network interface? What other steps can I take to get to the bottom of this problem? Thanks a lot