Re: Rewritten TCP reassembly

From: Andre Oppermann (andre_at_freebsd.org)
Date: 12/11/04

  • Next message: Andrew Gallatin: "Re: Rewritten TCP reassembly" 
    Date: Sat, 11 Dec 2004 00:20:37 +0100
To: Andrew Gallatin <gallatin@cs.duke.edu>

    Andrew Gallatin wrote:
    >
    > Andre Oppermann writes:
    > > I've totally rewritten the TCP reassembly function to be a lot more
    > > efficient. In tests with normal bw*delay products and packet loss
    > > plus severe reordering I've measured an improvment of at least 30% in
    > > performance. For high and very high bw*delay product links the
    > > performance improvement is most likely much higher.
    > >
    >
    > I ran netperf with 20 times for each of 3 socket buffer sizes (128KB,
    > 256KB, and 1MB) before and after patching. All tests were run with
    > net.isr.enable=1, and machdep.cpu_idle_hlt=0. CPU was pretty much
    > maxed thoughout. (SMP kernel, 1 HTT p4).

    HTT is not really good with FreeBSD's schedulers. Most of the time
    it hurts performance more than it helps.

    > I'm not sure how much weight you should give these result. I still
    > haven't gotten around to fixing kttcp, so the memory copy overhead is
    > pretty high. For xmit, b/w goes from 2.90Gb/sec to 3.95Gb/sec when
    > sendfile is used. (linux receiver on same hw).

    Ok.

    > I've also not gotten around to making any of Robert's suggested driver
    > optimizations.

    Ok.

    > I noticed that at least systat was not printing the number of tcp
    > out-of-order packets after applying the patch. I forgot to check
    > netstat itself.

    Duh, that's missing in the patch. Good you noticed.

    Regarding your measurements, did you measure the bandwidth as reported
    by Netperf? Is a FreeBSD box on both sides (you mentioned Linux)? 

    -- 
Andre
> Drew
> 
> x before.131072
> + after.131072
> +--------------------------------------------------------------------------+
> |                    ++  x  +                        x                    x|
> |+    *+  +        + ++ +xx++  x +x*  ++ +  xx+x x   x +  x     x      xxxx|
> |           |_____________AM|___________|_______AM__________________|      |
> +--------------------------------------------------------------------------+
>     N           Min           Max        Median           Avg        Stddev
> x  20       3336.63       3516.95       3449.87      3447.838     52.708815
> +  20          3323       3466.34       3391.97      3389.686     37.154513
> Difference at 95.0% confidence
>         -58.152 +/- 29.1859
>         -1.68662% +/- 0.846499%
>         (Student's t, pooled s = 45.5998)
> 
> x before.262144
> + after.262144
> +--------------------------------------------------------------------------+
> |            +                                                             |
> |            +                                            x                |
> |+      ++ + ++                                        x  x      x         |
> |+   + ++++++++      +           +         x  x x x    x xx xxx  xxxx x   x|
> |   |______A______|                                |_______AM_______|      |
> +--------------------------------------------------------------------------+
>     N           Min           Max        Median           Avg        Stddev
> x  20       3197.77       3421.67       3316.64     3314.8555     59.787371
> +  20       2889.64       3122.17       2965.01     2964.9865     50.076867
> Difference at 95.0% confidence
>         -349.869 +/- 35.2961
>         -10.5546% +/- 1.06479%
>         (Student's t, pooled s = 55.1463)
> 
> x before.1048576
> + after.1048576
> +--------------------------------------------------------------------------+
> |                 x                                    +                   |
> |xx      x      + x    xx  +  x+x++x    *  x*   x+*x+x+*    x*+  +  + +  ++|
> |               |_________________|A____M__________A__M____________|       |
> +--------------------------------------------------------------------------+
>     N           Min           Max        Median           Avg        Stddev
> x  20       2658.05       2732.45       2706.91     2700.1955     23.318833
> +  20       2676.13       2748.97       2724.07     2719.9665     20.531036
> Difference at 95.0% confidence
>         19.771 +/- 14.0613
>         0.732206% +/- 0.52075%
>         (Student's t, pooled s = 21.9692)
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
  • Next message: Andrew Gallatin: "Re: Rewritten TCP reassembly"