Re: tuning for high connection rates

Mike Tancsa wrote:
At 12:23 PM 12/5/2007, Philipp Wuensche wrote:
Mike Tancsa wrote:
At 07:14 PM 12/4/2007, Philipp Wuensche wrote:

The debug output of em0 looks like this:

em0: CTRL = 0x40140248 RCTL = 0x8002
em0: Packet buffer = Tx=20k Rx=12k
em0: Flow control watermarks high = 10240 low = 8740
em0: tx_int_delay = 66, tx_abs_int_delay = 66
em0: rx_int_delay = 32, rx_abs_int_delay = 66
em0: fifo workaround = 0, fifo_reset_count = 0
em0: hw tdh = 183, hw tdt = 183
em0: hw rdh = 139, hw rdt = 139
em0: Num Tx descriptors avail = 223
em0: Tx Descriptors not avail1 = 6225
em0: Tx Descriptors not avail2 = 3
em0: Std mbuf failed = 0
em0: Std mbuf cluster failed = 0
em0: Driver dropped packets = 0
em0: Driver tx dma failure in encap = 0

If you do a
sysctl -w dev.em.0.stats=1

It will spit the nic stats to syslog. What are the results ?

em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 12876719
em0: Receive No Buffers = 5950326
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 56256
em0: watchdog timeouts = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 0
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 3384408375
em0: Good Packets Xmtd = 2657550034
em0: TSO Contexts Xmtd = 6925441
em0: TSO Contexts Failed = 0


Also,
what does ifconfig em0 look like (i.e. what options do you have set,
speed etc)

Its on a 100Mbit/s switch, we haven't changed the options:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
media: Ethernet autoselect (100baseTX <full-duplex>)

Some people have reported that TSO is a "bad thing" on 100Mb. Can you
try disabling that ? Also, you seem to have a lot of RX overruns and
missed packets such that the nic cannot process things fast enough. I
havent done any benchmarks yet, but the Yandex people claim their
modified EM driver can handle higher PPS rates than the stock em
driver. Not sure if they have a RELENG_7 port or not but they might
have some insight.

After switching to net.isr.direct=0 and 346609775 good packets later, RX
overruns haven't increased by one! Thats nice. Still interrupt is using
up the CPU. I'm not quite sure if polling would help now!?

current stats:
em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 12885592
em0: Receive No Buffers = 5953357
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 56296
em0: watchdog timeouts = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 0
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 3731018150
em0: Good Packets Xmtd = 2936659631
em0: TSO Contexts Xmtd = 7734211
em0: TSO Contexts Failed = 0

We will try disabling TSO to see if anything changes.

greetings,
cryx

_______________________________________________
freebsd-performance@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"

Relevant Pages