Re: em forwarding performance (was Proposed 6.2 em RELEASE patch


I am moving this thread over to performance after this post as it makes more sense to continue there.



At 11:54 PM 11/19/2006, Mike Tancsa wrote:
At 04:30 PM 11/13/2006, Scott Long wrote:
Mike Tancsa wrote:
At 12:15 AM 11/13/2006, Scott Long wrote:

Is this with EM_INTR_FAST enabled also?

Without it, the 2 streams are definitely lossy on the management interface

---Mike

Ok, and would you be able to test the polling options as well?


Note about platforms. The HEAD w Patch is a patch glebius@xxxxxxxxxxx asked me to test. FastFWD is with net.inet.ip.fastforwarding on. Also with FastFWD set to one, I always used the kernel options ADAPTIVE_GIANT commented out and added NO_ADAPTIVE_MUTEXES. INET6 was removed from all kernels as well. With these kernel changes, and fast forwarding on, I was able to keep the box r2 responsive from the console as while blasting packets across its 2 interfaces. Otherwise, the box seemingly livelocked. For the linux kernel config, it was pretty well the default, except I removed INET6, IPSEC and disabled iptables. The LINUX kernel was 2.6.18.2 on FC5.

The first test is with UDP netperf.
/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t UDP_STREAM -- -m 10 -s 32768 -S 32768
/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t UDP_STREAM -- -m 64 -s 32768 -S 32768
/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t UDP_STREAM -- -m 128 -s 32768 -S 32768
/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t UDP_STREAM -- -m 200 -s 32768 -S 32768



Again, this is the same setup as described at http://www.tancsa.com/blast.jpg

My goals of this testing was to understand the following :

1) the new em driver to make sure it works well for me and give it a good shake out under load for RELENG_6

2) understand the implications of various configs on forwarding performance of SMP vs UP vs Polling vs Fast Interrupts and to avoid livelock when there is a lot of pps

In this round of testing, I tried of RELENG_6 i386 in UP config as well. Although raw packets / second (pps) forwarding was faster, the box was pretty unresponsive and userland apps (i.e. routing) and made the config pretty unusable with fast_forwarding enabled. Once ipfw was loaded, the box would totally lock up and routing daemons was start to spin out of control as hold timers expired. Bottom line, there is slightly less raw pps performance out of an SMP config, but the box seems to be far more resilient against a high pps attack.

RELENG_6 SMP with #define EM_FAST_INTR 1 in if_em.c
net.inet.ip.fastforwarding=1
with ADAPTIVE_GIANT removed from the kernel and
NO_ADAPTIVE_MUTEXES added

gives decent pps forwarding rates, without the box becoming live locked. Note, in the real world, given an average packet size of ~600 bytes, a box in this config should be able to route and firewall gigabit speeds without much issue and can sustain moderate DDoS pps attacks over the net.

For the routing test, I used 2 peers each sending ~ 195K routes.

I re-tested the single UDP stream with 194k routes loaded in the kernel routing table and 2 bgp peers. Then, while blasting across the box, I cleared the session which had all the routes installed in the kernel so that the daemon would have to reinstall all the routes to point to the other peer. While this was happening (10 seconds on SMP boxes, MUCH longer ~ 1min on UP, sometimes total failure) I was measuring throughput. On UP it didnt drop too much, a bit more on SMP, but convergence was quite fast, about 10 seconds. Similarly, installing ipfw rules on the UP version made the box totally live lock in non polling mode, but seemed to perform well enough in polling mode. pf lagged quite far behind

The biggest difference seems to be in the efficiency of the firewall rules in LINUX vs FreeBSD. Granted, the rules I inserted, are poorly written, but adding rules did seem to have a linearly negative impact on performance, where as rules via iptables did not significantly impact forwarding rates. However, in the LINUX test, I seemed to trigger some race in bgpd when doing the clear that took a little more proding with FreeBSD, but is there as well :( So it might be back to version .98

The table is also up at http://www.tancsa.com/blast.html which might be easier to read



Straight Routing test One Stream 194K Routes bgp clear and single ipfw 5 ipfw ruipfw 10 pf 1 pf 5
Linux 581,310 590,584 579,833 582,963 575,718 579,442
FreeBSD HEAD Nov 20 +FastFWD 540,695 529,425 439,980 398,283 370,458
FreeBSD HEAD Nov 11 441,560
RELENG6 i386 407,403
RELENG6 i386 FastFWD 557,580 562,547 484,250 425,791 386,644 353,856 333,293
FreeBSD HEAD w Patch 422,294
FreeBSD HEAD w Patch FastFWD 567,290 564,340 482,093 436,205 381,459 359,248 361,458
AMD64 RELENG6 w FastFWD 574,590 549,233 486,737 400,050 320,129 296,760 273,824
AMD64 RELENG6 polling 285,917
AMD64 RELENG6 polling FastFWD 512,042
RELENG6 i386 polling FastFWD 558,600 550,041
RELENG6 i386 polling FastFWD HZ=2000 565,520 563,068 373,904
RELENG_6 UP i386 400,188
RELENG_6 UP i386 FastFWD 584,900 582,534 560,033 560,323
RELENG_6 UP i386 FastFWD Polling 585,934 476,629 422,633 393,301


Straight Routing test 2 streams opposite direction
Linux 473,810 452,205 408,932
FreeBSD HEAD Nov 11 204,043
FreeBSD HEAD nov 20 + fastFWD 312,140 250,277 223,289 208,551
RELENG6 i386 165,461
RELENG6 i386 FastFWD 368,960 353,437 216,597 206,077 194,47669,50067,385
FreeBSD HEAD w Patch 127,832
FreeBSD HEAD w Patch FastFWD 346,220 404,785 249,617 234,047157,603
AMD64 RELENG6 w Polling 155,659
AMD64 RELENG6 w Polling FastFWD 231,541
RELENG6 i386 polling FastFWD 319,350 312,621
RELENG6 i386 polling FastFWD HZ=2000 300,280 299,018
RELENG_6 UP i386 FastFWD Polling 342,551 229,652 205,322



_______________________________________________
freebsd-net@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@xxxxxxxxxxx"

Relevant Pages