freebsd router project. Problems with polling?

• Previous message: Robert Watson: "Re: FreeBSD 5.3 I/O Performance / Linux 2.6.10 | Continued Discussion"
• Next in thread: Gleb Smirnoff: "Re: freebsd router project. Problems with polling?"
• Reply: Gleb Smirnoff: "Re: freebsd router project. Problems with polling?"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

To: freebsd-performance@freebsd.org
Date: Fri, 28 Jan 2005 18:18:19 +0100

Hello

Overview:

ATM, I've a 4.10 router (xeon 2.4ghz UP, dual on board em gigE
interfaces) in a productive enviroment. The routing software is quagga.
It's doing quite well.
It has an average of 200k pakets per seconds with the full routing
table. The System has polling enabled and interrupt is almost idle and
cpu too.

New one:
Now I'm building a FreeBSD 5.3 or -Stable based router system on CF
cards too. Intel whitepaper shows a max pps for the em cards at 1.4mpps.
I doubt that I can reach more than ~600k pps with the fully routing
table loaded! But this would be ok great.

My system: Xeon 2.4Ghz UP, 2x onboard em GigE interface, 5.3-RELEASE-p4
with if_em.c,v 1.44.2.4.

my configs:

sysctl.conf:
vm.swap_enabled=0
kern.ipc.somaxconn=1024
kern.polling.enable=1
kern.random.sys.harvest.ethernet=0
kern.random.sys.harvest.interrupt=0
net.inet.udp.recvspace=65536
net.inet.tcp.sendspace=65536
net.inet.tcp.delayed_ack=0
net.inet.tcp.msl=5000
net.inet.ip.maxfragpackets=40
net.inet.ip.maxfragsperpacket=4
net.inet.ip.fw.one_pass=0
net.inet.ip.fw.dyn_max=8192
net.inet.ip.fw.dyn_udp_lifetime=15
net.inet.ip.fastforwarding=1

loader.conf:
kern.ipc.nmbclusters=32768

Kernel config:
http://www.bsdunix.ch/public/ROUTER5

Setup:

I've 3 similar machines.
I use my own udpsend.c and udprec.c to send packages and to count the
pakages. It can send up to 300-330k pps (udp size 64 bytes)

ATM, the full routing table is not loaded. It's a very basic setup. The
goal is to find the maximum pps throughput for the router with small
pakets.
But atm I've problems with device polling.

Graphic:
 ------------------
|10.0.1.2 udp send |
 ------------------
        |
        |
-------em0------
|freebsd router |
-------em1------
        |
        |
-----------------------
| 192.168.1.2 udp recv |
------------------------

netstat -w 1 (polling disabled)
            input (Total) output
   packets errs bytes packets errs bytes colls
    300531 0 23441438 300585 0 23444792 0
    300939 0 25872898 300895 0 25870184 0
    300738 0 23457584 300768 0 23460626 0
    300304 10 25826466 300304 0 25826876 0
    ...

Interrupt load is about 70% with net.inet.ip.fastforwarding enabled. If
I disable this the system becomes unusable.
The system hasn't reach the limit yet. But the interrupt is much to
high. It's not worthy to add a second udp sender machine, at the moment.

netstat -w 1 (polling enabled)
            input (Total) output
   packets errs bytes packets errs bytes colls
    150151 47647 12910330 150150 0 12911806 0
    150151 0 11711798 150152 0 11711876 0
    150151 47665 12910986 150151 0 12911810 0
    150151 0 11711798 150151 0 11711798 0
    ...

Interrupt load is about 10%. CPU is about 60% and with
kern.polling.idle_poll enabled it goes to 100% (as expected).

As you see the speed is droping down to 50% with polling enable and and
I got a lot of errors.
kern.polling.lost_polls: 188748 and kern.polling.suspect: 186919 are
also very high. I don't know why polling is so bad on this machine. All
SMP option are disabled in the kernel and bios.
I tried to do as much as in
http://lists.freebsd.org/pipermail/freebsd-questions/2004-November/064427.html described.

I will prepare others tests with -STABLE and -CURRENT in the next few
days. At the mean time, are they some other magic things config option I
can try? Perhaps increase the HZ to 2000 in the kernel or remove polling
and try smp machine? I doubt that I can run the machine without polling.
If you see 70% interrupt load with 300k pps without polling.

regards
Thomas Vogt

_______________________________________________
freebsd-performance@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

• Previous message: Robert Watson: "Re: FreeBSD 5.3 I/O Performance / Linux 2.6.10 | Continued Discussion"
• Next in thread: Gleb Smirnoff: "Re: freebsd router project. Problems with polling?"
• Reply: Gleb Smirnoff: "Re: freebsd router project. Problems with polling?"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]