TCP Performance advice needed [long!]

I'm writing a simple client-server application and I think I have a performance problem. The basic communication between the server and the client is like this: the server sends messages at most MAX_SIZE in length, one by one, and waits until the client acknowleges each message before sending the next one. Messages of maximum length (MAX_SIZE) make about 75% or more of total messages. Client responses are small and of fixed size (<10 bytes). I have TCP_NODELAY turned on on both. All sends are done with a blocking send() call of required size. Client's response processing time doesn't depend notably on the message size.

I've done some simple benchmarks and I'm puzzled about the results. In the benchmarks, the MAX_SIZE is periodically increased from 256 to 65536 in power-of-2 steps. In the benchmark setup the server and client are connected directly via 100Mbps connection without transmission problems (FTP pushes ~12MB/s in both directions).

Problem: It seems that the packets-per-second rate sent by the server increases from 1000 at MAX_SIZE=256 to 2500 at MAX_SIZE=4096, then sharply drops to 400 at MAX_SIZE=8192 and above. Remembering a discussion on Samba performance, I tried disabling tcp.inflight.enable, and the results for large packet sizes became very good, about 4000pps, until MAX_SIZE=32768, where it drops to 400 again. What does tcp.inflight do, and can it be done without changing sysctls?

Another problem: the starting rates, e.g. ~1000pps at MAX_SIZE=256, seem too low to me - this pushes only 256kB/s. This is a problem because in real usage such small sizes could be common. Looking at netstat, packet rates seem to vary randomly. Here's netstat log for session with MAX_SIZE=512:


1 0 66 1 0 178 0
378 0 27956 378 0 132192 0
1405 0 103962 1405 0 488770 0
1193 0 88274 1193 0 414994 0
1093 0 80874 1093 0 380194 0
1010 0 74732 1011 0 351658 0
930 0 68812 930 0 323732 0
797 0 58970 796 0 276576 0
95 0 7022 95 0 32890 0
input (Total) output
packets errs bytes packets errs bytes colls
201 0 14866 201 0 69778 0
495 0 36622 495 0 172202 0
739 0 54678 739 0 257002 0
1291 0 95526 1292 0 449728 0
1134 0 83908 1133 0 394094 0
1026 0 75916 1026 0 356616 0
987 0 73030 987 0 343306 0
873 0 64594 873 0 303634 0
855 0 63262 855 0 297370 0
360 0 26632 360 0 125372 0
1002 0 74140 1002 0 348264 0
1166 0 86276 1166 0 405860 0
1056 0 78136 1056 0 367056 0
937 0 69330 937 0 325906 0
753 0 55714 753 0 261874 0
111 0 8206 111 0 38458 0
143 0 10566 144 0 49692 0
1 0 66 1 0 178 0
1 0 66 1 0 178 0

Here's for MAX_SIZE=8192:

input (Total) output
packets errs bytes packets errs bytes colls
1082 0 74292 1438 0 1580040 0
1287 0 88366 1701 0 1885062 0
1403 0 96342 1867 0 2044074 0
1763 0 121046 2346 0 2578684 0
1532 0 105208 2026 0 2236048 0
1866 0 128124 2463 0 2726554 0
2401 0 164874 3192 0 3508384 0
2797 0 192058 3719 0 4095824 0
2791 0 191646 3714 0 4070142 0
2773 0 190410 3692 0 4052512 0
2583 0 177358 3438 0 3780296 0
2947 0 202358 3927 0 4307032 0
2963 0 203462 3943 0 4324596 0
2946 0 202292 3926 0 4307032 0
2965 0 203594 3946 0 4341638 0
2989 0 205242 3977 0 4359552 0
2928 0 201056 3898 0 4289006 0
2995 0 205654 3983 0 4368192 0
2965 0 203594 3945 0 4333018 0
2983 0 204830 3973 0 4359598 0
2978 0 204484 3968 0 4359380 0
input (Total) output
packets errs bytes packets errs bytes colls
2986 0 205028 3976 0 4359910 0
2979 0 204566 3969 0 4359578 0
2123 0 145790 2827 0 3089432 0

It seems that the transmission for MAX_SIZE=8192 has slow start but I don't know how to disable it?

Help on any of these points, or suggestions where to investigate further will be appreciated!
_______________________________________________
freebsd-net@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Printer Driver for Windows 2003 Clustered Server
    ... Printer drivers installed on NT4.0 client from the Windwos 2003 Clustered ... server has a serious performance problem with displaying the Properties ... Samething happens to opening the Printing Preferences page. ... server and NT4.0 client environment. ...
    (microsoft.public.development.device.drivers)
  • Re: ThreadPool and IoCompletionPorts
    ... > what the server (in an example where the server receives and the ... > Perfectly innocent code that does nothing but receive from the socket. ... > client closes the connection ) this loop keeps on going ... This is where my performance problem lied, ...
    (microsoft.public.dotnet.framework.performance)
  • Re: TCP Performance advice needed [long!]
    ... protocol, that expects a reply before sending the next piece of data, will ... The basic communication between the server and the client is like this: the server sends messages at most MAX_SIZE in length, one by one, and waits until the client acknowleges each message before sending the next one. ... packets errs bytes packets errs bytes colls ...
    (freebsd-net)
  • Re: What doesnt lend itself to OO?
    ... >> proxy and instructs the server to constuct the real object. ... rather than client code. ... If 'clock' is instantiated in the server, ... > for the server interface at the OOA level. ...
    (comp.object)
  • This is going straight to the pool room
    ... or not the client has privilege to do what they're trying to do, ... The server environment is this: ... 3GL User action Routines that Tier3 will execute on your behalf during the ... Routine Name: USER_INIT ...
    (comp.os.vms)