Re: 64K limit for socket send

Dave Froble <davef@xxxxxxxxxxxxx> wrote:

In the socket communications applications I've designed, the following
happens:

0) service is listening for connection requests
1) client requests connection
2) Service grants connection
3) Client sends request
4) Service performs whatever, and sends data
5) client receives data and closes the connection
6) service goes back to listening for connection requests

If the server is the first to initiate close then that is an implicit
flush of any data queued awaiting Nagle. That is what things are/were
like for HTTP 1.0. So, that would be:

4) Service performs whatever, and sends data
4.a) Service performs shutdown(SHUT_WR)

4.b) Client receives last byte of data and the shutdown notification
(read return of zero triggered by the receipt of the server's
FINishied segment)

5) Client calls close()
6) Service receives the indication of client close with a zero-byte read
7) Serice calls close()
etc etc

It fits the application requirements well. Not much call for multiple
requests and replies from the client. I'm aware that there could be
applications that differ widely from such a design.

So my question is, say the data is 390,000 bytes, and my buffer used
within the application is 25,000 bytes. The applications have send and
receive functions that will send/receive until all data is
sent/received. I declare data size at the front of the data stream to
insure all is sent/received, and to know when no more is to be expected.

The send function will keep parsing off 25K bytes and sending them in a
loop until all the data is sent. The receive function will read until
it has the required number of bytes. This seems to work well. For more
than a few years anyway.

Now you're making me wonder whether I've introduced unneeded and
unwanted delays into the transmissions.

In many respects it is a race. And often as not, if the total
transfer time is rather longer than the receiver's standalone ACK
timer, any delay for a last, sub-MSS send (and the send must be
sub-MSS to hit this) might be epsilon. And it may not be a standalone
ACK - the data stream could be such that the last full-MSS segment
before the last sub-MSS send was enough to induce a window update or
an immediate ACK from the reciever.

Broad handwaving of when ACKs are sent:

a) piggy-backed on data goin the other way
b) piggy-backed on window updates going the other way
c) after the standalone ACK timer expires (50 to 200 milliseconds
depending on the stack)

Once I finish sending data, how long will TCP/IP take to send the last
partial buffer. Note that I'm assuming that my buffer and that used
internal to TCP/IP will not be the same size.

_IFF_ the last send is sub-MSS it will take no longer than it takes
for the remote reciever to send an ACK, which will depend on the
above.

I sort of like it better as a 'black box' that just works, rather than
having to always customize the size of the sends.

I've been hesitant to mention it because it is soooo often abused as a
way to kludge around application bugs, but there is the TCP_NODELAY
setsockopt() which will disable the Nagle algorithm for a connection.

The abuse is generally when applications send "logically associated"
data in separate calls to send() - the example I often give is an
email client sending the email headers separately from the email data.
For a more contemporary exmaple replace "email" with "http."

Typically in such applications, both the app headers and data are
small and likely to be affected by Nagle. The proper fix then is to
use a gathering write() like writev() or sendmsg(). Lazy programmers
will sometimes just set TCP_NODELAY. They see this big increase in
wall-clock perf but don't realize they are sending many more segments
than they should and so consuming more CPU than they should.

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...
.

Relevant Pages

  • [PATCH 0/5] [RFC] AF_RXRPC socket family implementation [try #3]
    ... These patches together supply secure client-side RxRPC connectivity as a Linux ... kernel socket family. ... presentation side is left to the client. ... Each connection goes to a particular "service". ...
    (Linux-Kernel)
  • Re: callbacks in TAO
    ... Most firewalls do not restrict access by inbound port number. ... Lets say your client application terminates while it still has the connection open. ... requests at the same time, the second thread will open a new connection if the existing connection is busy. ...
    (comp.object.corba)
  • [PATCH 0/5] [RFC] AF_RXRPC socket family implementation
    ... These patches together supply secure client-side RxRPC connectivity as a Linux ... Make it possible for the client socket to be used to go to more than one ... Each connection goes to a particular "service". ...
    (Linux-Kernel)
  • [PATCH 0/5] [RFC] AF_RXRPC socket family implementation [try #2]
    ... These patches together supply secure client-side RxRPC connectivity as a Linux ... Make it possible for the client socket to be used to go to more than one ... Each connection goes to a particular "service". ...
    (Linux-Kernel)
  • Re: Learning how to use "Remote Login"
    ... for client connection requests. ... "listening", by your definition. ... who listens for client requests. ...
    (comp.sys.mac.system)