Re: Routing SMP benefit

Hi all

Where are 'those numbers' you are referring to? have I missed some message
in the thread?


Tiffany Snyder wrote:
Hi Andre,
are those numbers for small (64 bytes) packets? Good job on pushing
the base numbers higher on the same HW.

What piqued my attention was the note that our forwarding
performance doesn't scale with multiple CPUs. Which means there's a lot of
work to be done :-) Have we taken a look at OpenSolaris' Surya
(http://www.opensolaris.org/os/community/networking/surya-design.pdf)
project? They allow multiple readers/single writer on the radix_node_head
(and not a mutex as we do) and we may be able to do the same to gain some
parallelism. There are other things in Surya that exploit multiple CPUs.
It's definitely worth a read. DragonFlyBSD seems to achieve parallelism by
classifying packet as flows and then redirecting the flows to different
CPUs. OpenSolaris also does something similar. We can definitely think along
those lines.

NOTE:
1) I said multiple instead of dual CPUs on purpose.
2) I mentioned OpenSolaris and DragonFlyBSD as examples and to acknowledge
the work they are doing and to show that FreeBSD is far behind and is losing
it's lustre on continuing to be the networking platform of choice.

Thanks,

Tiffany.


On 12/29/05, Andre Oppermann <andre@xxxxxxxxxxx > wrote:

Markus Oestreicher wrote:
Currently running a few routers on 5-STABLE I have read the
recent changes in the network stack with interest.
You should run 6.0R. It contains many improvements over 5-STABLE.

A few questions come to my mind:

- Can a machine that mainly routes packets between two em(4)
interfaces benefit from a second CPU and SMP kernel? Can both
CPUs process packets from the same interface in parallel?
My testing has shown that a machine can benefit from it but not
much in the forwarding performance. The main benefit is the
prevention of lifelock if you have very high packet loads. The
second CPU on SMP keeps on doing all userland tasks and running
routing protocols. Otherwise your BGP sessions or OSPF hellos
would stop and remove you from the routing cloud.

- From reading the lists it appears that net.isr.direct
and net.ip.fastforwarding are doing similar things. Should
they be used together or rather not?
net.inet.ip.fastforwarding has precedence over net.isr.direct and
enabling both at the same doesn't gain you anything. Fastforwarding
is about 30% faster than all other methods available, including
polling. On my test machine with two em(4) and an AMD Opteron 852
(2.6GHz) I can route 580'000 pps with zero packet loss on -CURRENT.
An upcoming optimization that will go into -CURRENT in the next
few days pushes that to 714'000 pps. Futher optimizations are
underway to make a stock kernel do close to or above 1'000'000 pps
on the same hardware.

--
Andre
_______________________________________________
freebsd-net@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to " freebsd-net-unsubscribe@xxxxxxxxxxx"

_______________________________________________
freebsd-net@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@xxxxxxxxxxx"




--


With best regards.

Hooman Fazaeli <fazaeli@xxxxxxxxxxx>
Sepehr S. T. Co. Ltd.






_______________________________________________
freebsd-net@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: Routing SMP benefit
    ... There are other things in Surya that exploit multiple CPUs. ... Can a machine that mainly routes packets between two em ... I can route 580'000 pps with zero packet loss on -CURRENT. ...
    (freebsd-net)
  • Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]
    ... wirespeed with a current x86 platform. ... with 64-byte packets and 64-header_size payloads. ... CPUs use is cache prefetching that is integrated with the network ... possible on Intel or AMD x86 CPUs we have a ceiling limited by RAM speed. ...
    (freebsd-net)
  • Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]
    ... with a current x86 platform. ... with 64-byte packets and 64-header_size payloads. ... CPUs use is cache prefetching that is integrated with the network ... possible on Intel or AMD x86 CPUs we have a ceiling limited by RAM speed. ...
    (freebsd-net)
  • Re: questions on NAPI processing latency and dropped network packets
    ... I've got an issue that's popped up with a deployed system running ... I'm looking for some help figuring out why incoming network ... After a recent userspace app change, we've started seeing packets being ... Interrupts are being processed on both cpus: ...
    (Linux-Kernel)
  • Re: 6.x, 4.x ipfw/dummynet pf/altq - network performance issues
    ... I'll be able to run some more basic tests tomorrow to see some results, but want to wrap my head around what's actually logically meant to be happening based on adjustments, etc. [I suspect this'll do nothing for the UDP issue, but at least I might be able to pipe some TCP traffic] ... Little packets with ip lengths of 28-29 bytes seem to do the most damage. ... UDP floods are much better handled - an ipfw block rule for the packet type and the machine responds as if there were no flood at all (until total bandwidth saturation or PPS limits of the hardware, which in this case was around 950Mbps). ...
    (freebsd-performance)