Re: TCP socket close problem

Staffan Ulfberg <staffan@xxxxxxxxxx> writes:

369 ESTABLISHED:drop (src=212.209.153.82,55478, dst=212.247.27.201,5000)3746edf6@8ed0b5b(win=ffff)<ACK> -> ESTABLISHED
rcv_nxt 3746edf6 rcv_wnd 100a4 snd_una 8ed0b5b snd_nxt 8ed27df snd_max 8ed27df
snd_wl1 3746edf6 snd_wl2 8ed0b5b snd_wnd ffff
370 ESTABLISHED:output (src=212.247.27.201,5000, dst=212.209.153.82,55478)[8ed16c3..8ed16af)@3746edf6(win=ffff)<ACK> -> ESTABLISHED
rcv_nxt 3746edf6 rcv_wnd 100a4 snd_una 8ed0b5b snd_nxt 8ed27df snd_max 8ed27df
snd_wl1 3746edf6 snd_wl2 8ed0b5b snd_wnd ffff
370 ESTABLISHED:drop (src=212.209.153.82,55478, dst=212.247.27.201,5000)3746edf6@8ed0b5b(win=ffff)<ACK,FIN> -> ESTABLISHED
rcv_nxt 3746edf6 rcv_wnd 100a4 snd_una 8ed0b5b snd_nxt 8ed27df snd_max 8ed27df
snd_wl1 3746edf6 snd_wl2 8ed0b5b snd_wnd ffff
370 ESTABLISHED:drop (src=212.209.153.82,55478, dst=212.247.27.201,5000)3746edf7@8ed110f<ACK,RST> -> ESTABLISHED
rcv_nxt 3746edf6 rcv_wnd 100a4 snd_una 8ed0b5b snd_nxt 8ed27df snd_max 8ed27df
snd_wl1 3746edf6 snd_wl2 8ed0b5b snd_wnd ffff

So, indeed, the packets with the FIN and RST (and also, a preceeding
ACK) are dropped by tcp_input. Need to figure out why...

I'm getting tired, so this is all speculation by now, but I was
reading tcp_input.c for a while and made a few observations:

The dropped RST is easy to explain, given then that the FIN was
dropped (from tcp_input.c):

* Note 2: Paul Watson's paper "Slipping in the Window" has shown
* that brute force RST attacks are possible. To combat this,
* we use a much stricter check while in the ESTABLISHED state,
* only accepting RSTs where the sequence number is equal to
* last_ack_sent. In all other states (the states in which a
* RST is more likely), the more permissive check is used.

Since the server has not acked the FIN packet, last_ack_sent was not
updated, and thus the RST is ignored.

This does not explain the dropped FIN, however.

I wrote before that this problem only occurs on "busy" sockets, with
unacked data on the way to the client. Actually, looking at the
dumps, it seems that there is always unacked data on the way to the
clients when the problem occurs.

So, it happens that on the problematic connections, the FIN is always
accompanied by an ACK for a sequence number that has already been
acked, while there is unacked data in transit. tcp_input makes sure
not to miss a window update in these cases, but to me it seems that a
FIN could very well be missed. If someone wants to look at the code,
it is just under the follwing comment in tcp_input:

/*
* In ESTABLISHED state: drop duplicate ACKs; ACK out of range
* ACKs. If the ack is in the range
* tp->snd_una < th->th_ack <= tp->snd_max
* then advance tp->snd_una to th->th_ack and drop
* data from the retransmission queue. If this ACK reflects
* more up to date window information we update our window information.
*/

The packet containing the duplicate packet is only dropped if it is
duplicate number 3 or 4, unless the sysctl "net.inet.tcp.rfc3042" is
set (which it is by default). If set, the code retransmits some data
and drops the packet even if it is the first or second duplicate.

Since I do not really have a clue about the tcp stack, some other part
of the code could of course make sure the FIN is not lost, but it
appears to me this is what is happening. I tried clearing the
net.inet.tcp.rfc3042 sysctl, and after way too few experiments to be
sure, it actually seems to reduce the number of hanging connections.

If this is what's happening, doesn't it look like a bug? BTW, this is
on 5.5-PRERELEASE.

Staffan
.

Relevant Pages

  • Re: Socket weirdness
    ... Anyway, if the application on one device does Shutdownthen if a packet containing data is received from the peer on that connection, then that is an not a valid packet and a packet with the RST bit set is sent clearing down the connection. ... In TCP there is simply *one* type of packet, this is unlike HDLC, which carries data in 'I' frames, has ACK frames which it calls 'RR', and lots more. ... And receiving a segment containing data is not valid where the local application has done shutdown). ...
    (microsoft.public.dotnet.framework)
  • Re: Wierd networking.
    ... it's using the latest ACK value from the client.. ... the client should have received the above packets and ACK for 3799410862 to include the FIN. ... Here the client ACK's the 1460-byte packet, ...
    (freebsd-net)
  • Re: establish connection without tcp options
    ... sequence 874312230 ...
    (freebsd-net)
  • Re: establish connection without tcp options
    ... sequence 874312230 ...
    (freebsd-net)
  • Re: How TCP handle (RST,SYN) at initial connection establishment
    ... > SYN) or (RST, ACK) or simply RST upon receiving of any ... > SYN request towards victim server. ... > acceptable if the ACK field acknowledges the SYN." ... The "client", during the three-way handshacking, starts in CLOSED state, ...
    (Security-Basics)