Re: serious networking (em) performance (ggate and NFS) problem

From: Robert Watson (rwatson_at_freebsd.org)
Date: 11/19/04

  • Next message: Herve Boulouis: "Re: fast data access mmu miss with 5.3-STABLE from today" 
    Date: Fri, 19 Nov 2004 12:56:05 +0000 (GMT)
To: Emanuel Strobl <Emanuel.Strobl@gmx.net>

    On Fri, 19 Nov 2004, Emanuel Strobl wrote:

    > Am Donnerstag, 18. November 2004 13:27 schrieb Robert Watson:
    > > On Wed, 17 Nov 2004, Emanuel Strobl wrote:
    > > > I really love 5.3 in many ways but here're some unbelievable transfer
    >
    > First, thanks a lot to all of you paying attention to my problem again.
    > I'll use this as a cumulative answer to the many postings of you, first
    > answering Roberts questions and at the bottom those of the others.
    >
    > I changed cables and couldn't reproduce that bad results so I changed
    > cables back but also cannot reproduce them, especially the ggate write,
    > formerly with 2,6MB/s now performs at 15MB/s, but I haven't done any
    > polling tests anymore, just interrupt driven, since Matt explained that
    > em doesn't benefit of polling in any way.
    >
    > Results don't indicate a serious problem now but are still about a third
    > of what I'd expected with my hardware. Do I really need Gigahertz Class
    > CPUs to transfer 30MB/s over GbE?

    Well, the claim that if_em doesn't benefit from polling is inaccurate in
    the general case, but quite accurate in the specific case. In a box with
    multiple NIC's, using polling can make quite a big difference, not just by
    mitigating interrupt load, but also by helping to prioritize and manage
    the load, preventing live lock. As I indicated in my earlier e-mail,
    however, on your system it shouldn't make much difference -- 4k-8k
    interrupts/second is not a big deal, and quite normal for use of an if_em
    card in the interrupt-driven configuration.
     
    It looks like the netperf TCP test is getting just under 27MB/s, or
    214Mb/s. That does seem on the low side for the PCI bus, but it's also
    instructive to look at the netperf UDP_STREAM results, which indicate that
    the box believes it is transmitting 417Mb/s but only 67Mb/s are being
    received or processed fast enough by netserver on the remote box. This
    means you've achieved a send rate to the card of about 54Mb/s. Note that
    you can actually do the math on cycles/packet or cycles/byte here -- with
    TCP_STREAM, it looks like some combination of recipient CPU and latency
    overhead is the limiting factor, with netserver running at 94% busy.

    Could you try using geom gate to export a malloc-backed md device, and see
    what performance you see there? This would eliminate the storage round
    trip and guarantee the source is in memory, eliminating some possible
    sources of synchronous operation (which would increase latency, reducing
    throughput). Looking at CPU consumption here would also be helpful, as it
    would allow us to reason about where the CPU is going.

    > I was aware of that and because of lacking a GbE switch anyway I decided
    > to use a simple cable ;)

    Yes, this is my favorite configuration :-).

    > > (5) Next, I'd measure CPU consumption on the end box -- in particular, use
    > > top -S and systat -vmstat 1 to compare the idle condition of the
    > > system and the system under load.
    > >
    >
    > I additionally added these values to the netperf results.

    Thanks for your very complete and careful testing and reporting :-).

    Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
    robert@fledge.watson.org Principal Research Scientist, McAfee Research

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

  • Next message: Herve Boulouis: "Re: fast data access mmu miss with 5.3-STABLE from today"

    Relevant Pages