Re: Test on 10GBE Intel based network card

Who ya gonna call, why me of course, its my driver :)

Hmmm, the numbers on those look bogus, like some uninitialized variables.
You did say you aren't using flow control, right?

Jack


On Mon, Aug 17, 2009 at 2:52 PM, Carlos Pardo <cpardo@xxxxxxxxxxxx> wrote:

Hi Jack,



Thanks for the quick response. We can not use LRO because of the way we
accelerate on the WAN ports. We just moved from 7.0 to 8.0 to use your
latest driver (1.8.8). One thing we do not understand in 8.0. We are having
insane numbers for XON/XOFF Rcvd counters with essentially no traffic.
Driver version 1.2.16 works fine. Who should we contact for help?



ix0: Std Mbuf Failed = 0

ix0: Missed Packets = 0

ix0: Receive length errors = 0

ix0: Crc errors = 0

ix0: Driver dropped packets = 0

ix0: watchdog timeouts = 0

*ix0: XON Rcvd = 7950055973552*

ix0: XON Xmtd = 0

*ix0: XOFF Rcvd = 7950055973552*

ix0: XOFF Xmtd = 0

ix0: Total Packets Rcvd = 2149

ix0: Good Packets Rcvd = 2149

ix0: Good Packets Xmtd = 1001

ix0: TSO Transmissions = 0

ix1: Std Mbuf Failed = 0

ix1: Missed Packets = 0

ix1: Receive length errors = 0

ix1: Crc errors = 0

ix1: Driver dropped packets = 0

ix1: watchdog timeouts = 0

*ix1: XON Rcvd = 7946320044993*

ix1: XON Xmtd = 0

*ix1: XOFF Rcvd = 7946320044993*

ix1: XOFF Xmtd = 0

ix1: Total Packets Rcvd = 1002

ix1: Good Packets Rcvd = 1002

ix1: Good Packets Xmtd = 1588

ix1: TSO Transmissions = 0



Regards,



C Pardo



*From:* Jack Vogel [mailto:jfvogel@xxxxxxxxx]
*Sent:* Friday, August 14, 2009 3:15 PM
*To:* Carlos Pardo
*Cc:* freebsd-performance@xxxxxxxxxxx
*Subject:* Re: Test on 10GBE Intel based network card



I've talked over the issues with the guy on our team who has been most
involved in 10G performance, he asserts that 3Gbs will saturate a single
cpu with a small packet size, this is why you need multiqueue across
multiple cores. He was dubious about the FIFO assertion, its a relative
thing, if you can keep the thing drained it won't be a problem, doing that
is a complex combination of factors, the cpu, the bus, the memory....

If you don't deal with the systemic issues just cuz you go from an 82598
to a 82599 is not going to solve things.

What about LRO, are/can you use that? I never saw an answer about the
forwarding question, you can't use LRO in that case of course.

Regards,

Jack

On Fri, Aug 14, 2009 at 2:33 PM, Carlos Pardo <cpardo@xxxxxxxxxxxx> wrote:

Hi Jack,

I have a quick question. We are getting too many missed packets per minute
running about 3Gbs traffic. We can not use frame control in our application.
We are assuming that there is no way to improve upon the problem since it
seems to be a hardware limitation with the receive FIFO. We are using the
Intel® 82598EB 10 Gigabit Ethernet Controller. When can we expect the next
generation card from Intel? Thanks for any information you may provide.

Typical error count "ix0: Missed Packets = 81174" after a few minutes.

Best Regards,

Cpardo



-----Original Message-----
From: owner-freebsd-performance@xxxxxxxxxxx [mailto:
owner-freebsd-performance@xxxxxxxxxxx] On Behalf Of Invernizzi Fabrizio

Sent: Wednesday, August 05, 2009 3:13 AM
To: Jack Vogel; Julian Elischer
Cc: freebsd-performance@xxxxxxxxxxx; Stefan Lambrev
Subject: RE: Test on 10GBE Intel based network card

No improvement with kern.ipc.nmbclusters=262144 and 1.8.6 driver :<(((((

++fabrizio

------------------------------------------------------------------
Telecom Italia
Fabrizio INVERNIZZI
Technology - TILAB
Accesso Fisso e Trasporto
Via Reiss Romoli, 274 10148 Torino
Tel. +39 011 2285497
Mob. +39 3316001344
Fax +39 06 41867287


________________________________
From: Jack Vogel [mailto:jfvogel@xxxxxxxxx]
Sent: martedì 4 agosto 2009 18.42
To: Julian Elischer
Cc: Invernizzi Fabrizio; freebsd-performance@xxxxxxxxxxx; Stefan Lambrev
Subject: Re: Test on 10GBE Intel based network card

Your nmbclusters is very low, you list it twice so I'm assuming the second
value is
what it ends up being, 32K :(

I would set it to:

kern.ipc.nmbclusters=262144

Also, I thought you were using the current driver, but now it looks like
you are
using something fairly old, use my latest code which is 1.8.8

Jack

On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer <julian@xxxxxxxxxxxx
<mailto:julian@xxxxxxxxxxxx>> wrote:
Invernizzi Fabrizio wrote:
The limitation that you see is about the max number of packets that
FreeBSD can handle - it looks like your best performance is reached at
64 byte packets?
If you are meaning in term of Packet per second, you are right. These
are the packet per second measured during tests:
64 byte: 610119 Pps
512 byte: 516917 Pps
1492 byte: 464962 Pps


Am I correct that the maximum you can reach is around 639,000 packets
per second?
Yes, as you can see the maximum is 610119 Pps.
Where does this limit come from?
ah that's the whole point of tuning :-)
there are severalpossibities:
1/ the card's interrupts are probably attache dto aonly 1 cpu,
so that cpu can do no more work

This seems not to be the problem. See below a top snapshot during a
64byte-long packet storm

last pid: 8552; load averages: 0.40, 0.09, 0.03
up
0+20:36:58 09:40:29
124 processes: 13 running, 73 sleeping, 38 waiting
CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle
Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free
Swap: 2048M Total, 2048M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle:
cpu3
14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle:
cpu0
12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle:
cpu2
13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle:
cpu1
42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq
38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq
44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq
40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq
....

It looks like the 4 rxq processes are bound to the 4 available cores with
equal distribution.



2/ if more than 1 cpu is working, it may be that there is a lock in
heavy contention somewhere.

This I think is the problem. I am trying to understand how to
1- see where the heavy contention is (context switching? Some limiting
setting?)
2- mitigate it



there ia a lock profiling tool that right now I can't remember the name
of..

look it up with google :-) FreeBSD lock profiling tool

ah, first hit...

http://blogs.epfl.ch/article/23832



is the machine still responsive to other networks while
running at maximum capacity on this network? (make sure that
the other networks are on a differnet CPU (hmm I can't remember how to
do that :-).



Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle
persone indicate. La diffusione, copia o qualsiasi altra azione derivante
dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora
abbiate ricevuto questo documento per errore siete cortesemente pregati di
darne immediata comunicazione al mittente e di provvedere alla sua
distruzione, Grazie.

This e-mail and any attachments is confidential and may contain privileged
information intended for the addressee(s) only. Dissemination, copying,
printing or use by anybody else is unauthorised. If you are not the intended
recipient, please delete this message and any attachments and advise the
sender by return e-mail, Thanks.

_______________________________________________

freebsd-performance@xxxxxxxxxxx<mailto:freebsd-performance@xxxxxxxxxxx>
mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "
freebsd-performance-unsubscribe@xxxxxxxxxxx<mailto:
freebsd-performance-unsubscribe@xxxxxxxxxxx>"


Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle
persone indicate. La diffusione, copia o qualsiasi altra azione derivante
dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora
abbiate ricevuto questo documento per errore siete cortesemente pregati di
darne immediata comunicazione al mittente e di provvedere alla sua
distruzione, Grazie.

This e-mail and any attachments is confidential and may contain privileged
information intended for the addressee(s) only. Dissemination, copying,
printing or use by anybody else is unauthorised. If you are not the intended
recipient, please delete this message and any attachments and advise the
sender by return e-mail, Thanks.

[cid:00000000000000000000000000000001@xxxxxxxxxxxxx]Rispetta l'ambiente.
Non stampare questa mail se non è necessario.



_______________________________________________
freebsd-performance@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"

Loading