Re: FreeBSD 8.0 ixgbe Poor Performance



I believe that "pciconf -lvc" showed that the cards were in the correct
slot. I'm not sure as to what all of the output means but I'm guessing
that " cap 10[a0] = PCI-Express 2 endpoint max data 128(256) link
x8(x8)" means that the card is an 8 lane card and is using all 8 lanes.

Setting kern.ipc.maxsockbuf to16777216 got a better result with ipref
TCP testing. The rate when from ~2.5Gpbs to ~5.5Gbps.

Running iperf in UDP test mode is still yielding ~2.5Gbps. Running
tcpreplay tests is also yielding ~2.5Gbps as well.

Command lines for iperf testing are:

ipref -t 10 -w 2.5m -l 2.5m -c 169.1.0.2
iperf -s -w 2.5m -B 169.1.0.2

iperf -t 10 -w 2.5m -c 169.1.0.2 -u
iperf -s -w 2.5m -B 169.1.0.2 -u

For the tcpdump test, I'm sending output to /dev/null and using the
cache flag on tcpreplay in order to avoid limiting my network interface
throughput to the disk speed.
Commands lines for this test are:

tcpdump -i ix1 -w /dev/null
tcpreplay -i ix1 -t -l 0 -K ./rate.pcap

Please forgive my lack of kernel building prowess but I'm guessing that
the latest driver needs to be built in a FreeBSD STABLE tree. I ran
into an undefined symbol "drbr_needs_enqueue" in the ixgbe code I
downloaded.

Thanks for all the help.

On 4/21/2010 4:53 PM, Jack Vogel wrote:
Use my new driver and it will tell you when it comes up with the slot
speed is,
and if its substandard it will SQUAWK loudly at you :)

I think the S5000PAL only has Gen1 PCIE slots which is going to limit you
somewhat. Would recommend a current generation (x58 or 5520 chipset)
system if you want the full benefit of 10G.

BTW, you dont way what adapter, 82598 or 82599, you are using?

Jack


On Wed, Apr 21, 2010 at 12:52 PM, Stephen Sanders
<ssanders@xxxxxxxxxxxxxx <mailto:ssanders@xxxxxxxxxxxxxx>> wrote:

I'd be most pleased to get near 9k.

I'm running FreeBSD 8.0 amd64 on both of the the test hosts. I've
reset
the configurations to system default as I was getting no where with
sysctl and loader.conf settings.

The motherboards have been configured to do MSI interrupts. The
S5000PAL has a MSI to old style interrupt BIOS setting that
confuses the
driver interrupt setup.

The 10Gbps cards should be plugged into the 8x PCI-E slots on both
hosts. I'm double checking that claim right now and will get back
later.

Thanks


On 4/21/2010 2:13 PM, Jack Vogel wrote:
> When you get into the 10G world your performance will only be
as good
> as your weakest link, what I mean is if you connect to something
that has
> less than stellar bus and/or memory performance it is going to
throttle
> everything.
>
> Running back to back with two good systems you should be able to get
> near line rate (9K range). Things that can effect that: 64 bit
kernel,
> TSO, LRO, how many queues come to mind. The default driver config
> should get you there, so tell me more about your hardware/os
config??
>
> Jack
>
>
>
> On Wed, Apr 21, 2010 at 8:04 AM, Brandon Gooch
> <jamesbrandongooch@xxxxxxxxx
<mailto:jamesbrandongooch@xxxxxxxxx>>wrote:
>
>
>> On Wed, Apr 21, 2010 at 9:32 AM, Stephen Sanders
>> <ssanders@xxxxxxxxxxxxxx <mailto:ssanders@xxxxxxxxxxxxxx>> wrote:
>>
>>> I am running speed tests on a pair of systems equipped with
Intel 10Gbps
>>> cards and am getting poor performance.
>>>
>>> iperf and tcpdump testing indicates that the card is running
at roughly
>>> 2.5Gbps max transmit/receive.
>>>
>>> My attempts at turning fiddling with netisr, polling, and
varying the
>>> buffer sizes has been fruitless. I'm sure there is something
that I'm
>>> missing so I'm hoping for suggestions.
>>>
>>> There are two systems that are connected head to head via
cross over
>>> cable. The two systems have the same hardware configuration. The
>>> hardware is as follows:
>>>
>>> 2 Intel E5430 (Quad core) @ 2.66 Ghz
>>> Intel S5000PAL Motherboard
>>> 16GB Memory
>>>
>>> My iperf command line for the client is:
>>>
>>> iperf -t 10 -c 169.0.0.1 -w 2.5M -l 2.5M
>>>
>>> My TCP dump test command lines are:
>>>
>>> tcpdump -i ix0 -w/dev/null
>>> tcpreplay -i ix0 -t -l 0 -K ./test.pcap
>>>
>> If you're running 8.0-RELEASE, you might try updating to 8-STABLE.
>> Jack Vogel recently committed updated Intel NIC driver code:
>>
>> http://svn.freebsd.org/viewvc/base/stable/8/sys/dev/ixgbe/
>>
>> -Brandon
>> _______________________________________________
>> freebsd-performance@xxxxxxxxxxx
<mailto:freebsd-performance@xxxxxxxxxxx> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>> To unsubscribe, send any mail to "
>> freebsd-performance-unsubscribe@xxxxxxxxxxx
<mailto:freebsd-performance-unsubscribe@xxxxxxxxxxx>"
>>
>>
> _______________________________________________
> freebsd-performance@xxxxxxxxxxx
<mailto:freebsd-performance@xxxxxxxxxxx> mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to
"freebsd-performance-unsubscribe@xxxxxxxxxxx
<mailto:freebsd-performance-unsubscribe@xxxxxxxxxxx>"
>
>



_______________________________________________
freebsd-performance@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"



Relevant Pages

  • [PATCH 1a/3] WAN Kconfig: change "depends on HDLC" to "select"
    ... config PCI200SYN ... tristate "Goramo PCI200SYN support" ... Driver for PCI200SYN cards by Goramo sp. ...
    (Linux-Kernel)
  • Re: VGA cards
    ... but people may need some functions that are not provided by Intel ... And remember that Intel driver also had its own glitches... ... So ATI are another brand, ... overall hassle is generally far less on using Intel cards. ...
    (Debian-User)
  • [2.6 patch] VIDEO_V4L1 shouldnt be user-visible
    ... config VIDEO_V4L1_COMPAT ... tristate "AV7110 cards" ... Support for SAA7146 and AV7110 based DVB cards as produced ...
    (Linux-Kernel)
  • Re: xorg.conf no longer changeabe by system-config-display
    ... I have an IBM T43 with the Intel 915 chipset running FC5. ... I tried and tried to configure dual head operation several times and with several different pci secondary video cards without much progress. ... This was on a computer with an Intel 815 built into the main board. ... 915GM chipset, but the screen is not updated correctly when I set up X to use the i815 driver, although it worked fine in FC4. ...
    (Fedora)
  • Re: FreeBSD 8.0 ixgbe Poor Performance
    ... Adding "-P 2 " to the iperf client got the rate up to what it should ... I believe that "pciconf -lvc" showed that the cards were in the ... that the latest driver needs to be built in a FreeBSD STABLE ... hardware/os config?? ...
    (freebsd-performance)