Re: bge loader tunables

On Sat, Nov 17, 2007 at 09:13:50PM +1100, Bruce Evans wrote:

On Sat, 17 Nov 2007, Igor Sysoev wrote:

On Sat, Nov 17, 2007 at 08:30:58AM +1100, Bruce Evans wrote:

On Fri, 16 Nov 2007, Igor Sysoev wrote:

The attached patch creates the following bge loader tunables:

I plan to commit old work to do this using sysctls. Tunables are
harder to use and aren't needed since changes to the defaults aren't
needed for booting. I also implemented dynamic tuning for rx coal
parameters so that the sysctls are mostly not needed. Ask for patches
if you want to test this extensively.

Yes, I can test your patches on 6.2 and 7.0.
Now bge set the coalescing parameters at attach time.
Do the sysctl's allow to change them on-the-fly ?
How does rx dynamic tuning work ?
Could it be turned off ?

OK, the patch is enclosed at the end, in 2 versions:
- all my patches for bge (with lots of debugging cruft and half-baked
fixes for 5705+ sysctls.
- edited version with only the coalescing parameter changes.

I haven't used it under 6.2, but have used a similar version in ~5.2,
and it should work in 6.2 except for the 5705+ sysctl fixes.

bge actually sets parameters at init time, and it initializes whenever the
link is brought back up, so the parameters can be changed using
"ifconfig bgeN down up". Several network drivers have interrupt moderation
parameters that can be changed in this way, but it is painful to change
the link status like that, so I have a sysctl dev.bge.N.program_coal to
apply the current parameters to the hardware. The other sysctls to change
the parameters don't apply immediately, except the one for the rx tuning
max interrupt rate, since applying the changed parameters to the hardware
takes more code than a SYSCTL_INT(), and it is sometimes necessary to
change all the parameters together atomically.

Dynamic tuning works by monitoring the current rx packet rate and
increasing the active rx_max_coal_bds so that the ratio <rx packet
rate> / rx_max_coal_bds is usually <= the specified max rx interrupt
rate. rx_coal_ticks is set to the constant value of the inverse of
the specified max rx interrupt rate (in ticks) on transition to dynamic
mode but IIRC is not changed when the dynamic rate is changed (not
always changing it automatically allows adjusting it independently of
the rate but is often not what is wanted). The transition has some
bias towards lower latency over too many interrupts, so that short
bursts don't increase the latency. I think this simple algorithm is
good enough provided the load (in rx packets/second) doesn't oscillate
rapidly.

Dynamic tuning requires efficient reprogramming of at least one of the
hardware coal registers so that the tuning can respond rapidly to changes.
I have 2 methods for this:
- bge_careful_coal = 1 avoids using uses a potentially very long
busy-wait loop in the interrupt handler by giving up on reprogramming
the host coalescing engine (HCE) if the HCE seems to be busy. Docs
seem to require waiting for up to several milliseconds for the HCE
to stablilize, and it is not clear if it is possible for the HCE to
never stabilize because packets are streaming in. (I don't have
proper docs.) This seems to always work (the HCE is never busy)
for rx_max_coal_bds, but something near here didn't work for
changing rx_coal_ticks in an old version.
- bge_careful_coal = 0 avoids the loop by writing to the rx_max_coal_bds
register without waiting for the HCE. This seems to work too. It
isn't critical for the HCE to see the change immediately or even
for it to be seen at all (missed changes might do more than give a
huge interrupt rate for too long), but it is important for the
change to not break the engine.
There is no sysctl for this of for some other hackish parameters. The
source must be edited to change this from 1 to 0.

Dynamic tuning is turned off by setting the dynamic max interrupt
frequency to 0. Then rx_coal_ticks is reset to 150, and the active
rx_max_coal_bds is restored to the static value.

Finally I have tested your second (without debug stuff) patch in
production environment (~45K in/out packets) on FreeBSD 7.0-STABLE.
I think it should be commited.

I use my usual static settings in /etc/sysctl.conf:

dev.bge.0.dyncoal_max_intr_freq=0
#
dev.bge.0.rx_coal_ticks=500
dev.bge.0.tx_coal_ticks=10000
dev.bge.0.rx_max_coal_bds=64
dev.bge.0.tx_max_coal_bds=128
# apply the above parameters
dev.bge.0.program_coal=0

and have about only 1700-1900 interrupts per second.

The only issue was at boot time:

dev.bge.0.dyncoal_max_intr_freq: 10000 -> 0
dev.bge.0.rx_coal_ticks: 0 -> 500
dev.bge.0.tx_coal_ticks: 1000000 -> 10000
dev.bge.0.rx_max_coal_bds: 128 -> 64
dev.bge.0.tx_max_coal_bds: 384 -> 128
...
bge0: flags =8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
...
Local package initialization:
...
dev.bge.0.rx_coal_ticks: 150 -> 500

When disabling dyncoal_max_intr_freq at bge UPing resets rx_coal_ticks to 150.


--
Igor Sysoev
http://sysoev.ru/en/
_______________________________________________
freebsd-net@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: bge loader tunables
    ... parameters so that the sysctls are mostly not needed. ... Several network drivers have interrupt moderation ... the host coalescing engine (HCE) if the HCE seems to be busy. ...
    (freebsd-net)
  • interrupt statistics
    ... ISTR there is a tool (other than systat -vmstat) that shows interrupt ... statistics for all interrupts, but I can't find anything except the ... hw.intrnames and hw.intrcnt sysctls, which aren't directly human- ...
    (freebsd-hackers)