Re: Intel 10Gb
- From: Alexander Sack <pisymbol@xxxxxxxxx>
- Date: Fri, 14 May 2010 09:32:08 -0400
On Tue, May 11, 2010 at 9:51 AM, Andrew Gallatin <gallatin@xxxxxxxxxxx> wrote:
Murat Balaban [murat@xxxxxxxxxxxxx] wrote:
Much of the FreeBSD networking stack has been made parallel in order to
cope with high packet rates at 10 Gig/sec operation.
I've seen good numbers (near 10 Gig) in my tests involving TCP/UDP
send/receive. (latest Intel driver).
As far as BPF is concerned, above statement does not hold true,
since there is some work that needs to be done here in terms
of BPF locking and parallelism. My tests show that there
is a high lock contention around "bpf interface lock", resulting
in input errors at high packet rates and with many bpf devices.
If you're interested in 10GbE packet sniffing at line rate on the
cheap, have a look at the Myri10GE "sniffer" interface. This is a
special software package that takes a normal mxge(4) NIC, and replaces
the driver/firmware with a "myri_snf" driver/firmware which is
optimized for packet sniffing.
Using this driver/firmware combo, we can receive minimal packets at
line rate (14.8Mpps) to userspace. You can even access this using a
libpcap interface. The trick is that the fast paths are OS-bypass,
and don't suffer from OS overheads, like lock contention. See
http://www.myri.com/scs/SNF/doc/index.html for details.
But your timestamps will be atrocious at 10G speeds. Myricom doesn't
timestamp packets AFAIK. If you want reliable timestamps you need to
look at companies like Endace, Napatech, etc.
We do a lot of packet capture and work on bpf(4) all the time. My
biggest concern for reliable 10G packet capture is timestamps. The
call to microtime up in catchpacket() is not going to cut it (it
barely cuts it for GIGE line rate speeds).
I'd be interested in doing the multi-queue bpf(4) myself (perhaps I
should ask? I don't know if non-summer-of-code folks are allowed?).
I believe the goal is not so much throughput but cache affinity. It
would be nice if say the listener application (libpcap) could bind
itself to the same core that the driver's queue is receiving packets
on so everything from catching to post-processing all work with a very
warm cache (theoretically). I think that's the idea.
It would also allow multiple applications to subscribe to potentially
different queues that are doing some form of load balancing. Again,
Intel's 82599 chipset supports flow based queues (albeit the size of
the flow table is limited).
Note, zero-copy bpf(4) is your friend in all use cases at 10G speeds! :)
PS I am not sure but Intel also supports writing packets directly in
cache (yet I thought the 82599 driver actually does a prefetch anyway
which had me confused on why that helps)
freebsd-net@xxxxxxxxxxx mailing list
To unsubscribe, send any mail to "freebsd-net-unsubscribe@xxxxxxxxxxx"
- Re: Intel 10Gb
- From: Andrew Gallatin
- Re: Intel 10Gb
- Prev by Date: Re: sockstat / netstat output 8.x vs 7.x
- Next by Date: Re: Intel 10Gb
- Previous by thread: Re: Intel 10Gb
- Next by thread: Re: Intel 10Gb