RE: TX Multiqueue?



-----Original Message-----
From: owner-freebsd-current@xxxxxxxxxxx
[mailto:owner-freebsd-current@xxxxxxxxxxx] On Behalf Of Jack Vogel
Sent: Monday, September 24, 2007 10:28 AM
To: Kip Macy
Cc: Darren Reed; freebsd-net@xxxxxxxxxxx; FreeBSD Current
Subject: Re: TX Multiqueue?

On 9/23/07, Kip Macy <kip.macy@xxxxxxxxx> wrote:
On 9/23/07, Darren Reed <darrenr@xxxxxxxxxxx> wrote:
Kip Macy wrote:
My ethng branch supports multiple rx and tx queues.

-Kip


What are your plans for how we use/manage/interact with the mutiple
rx/tx queues?

The rx hardware queue is determined by the hardware. Different
hardware allows for different policies. I just use the stock rss_hash
of a crc32 of the 4-tuple in cxgb. I've added a field to the pkthdr
which cxgb uses the least significant bits of to determine which
outbound queue to use. Its up to the upper layers to determine how to
set those bits. One of the changes that is required to take advantaged
of this is moving the queues into the driver. I've added a new
if_start function to ifnet to take advantage of this. I also have a
normal if_start function for backward compatibility.

Yes, the queues in Oplin and Zoar are also a hardware feature, not
just some software infrastructure. There are a number of different
ways that it can be configured. I did not have some settled notion
of how things should be managed but I would rather not have it
be something done under the covers in the driver, a configurable
stack option seems better to me. This needs to be done right or
it will just be a hack, I don't know who the right parties are, it
should
not be just a one-person decision, those with a stake in this sort
of thing should all be involved.

[Muhammad Shafiq]
Based upon our experience, form other OS(s), the HW TX/RX queues are
quite helpful in reducing CPU utilization (in MP machines). For example,
XFRAME-I/II TX/RX queues, originally intended for "I/O Virtualization",
can be affiliated to specific CPU(s); the resulting locality of
reference improves cache hits and reduces lock contention, if any. The
HW details of this mechanism can be found at the following link:
http://www.neterion.com/support/xframe_developer.html

We would prefer to have a configurable, generic, mechanism to exploit
the multiple TX/RX queues, now supported by NIC vendors. If possible,
the mechanism should be extendable to fit into specific
capability/requirement of HW vendors.




Cheers,

Jack
_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to
"freebsd-current-unsubscribe@xxxxxxxxxxx"
_______________________________________________
freebsd-net@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: svn commit: r180256 - head/sys/dev/arl
    ... IFQ_MAXLEN is also too small for 1Gbps or even 100Nbps hardware devices, so only drivers for old hardware and some software drivers can use it anyway. ... And, just to put it on the table in e-mail, since I know it has come up a lot at developer summits: the ALTQ infrastructure is decreasingly compatible with current network devices, which often have quite large queues in hardware, or where there are multiple transmit queues. ... One possibility I've been considering is making the whole ifq subsystem a library to device drivers, rather than a required interface to transmit. ... This would allow the device driver to instantiate more than one if there are multiple hardware queues that need to be represented, or, for example, allow synthetic encapsulation interfaces to avoid queueing entirely and directly dispatch to the lower layer interface without requiring a mandatory enqueue/dequeue step. ...
    (freebsd-net)
  • Re: TX Multiqueue?
    ... The rx hardware queue is determined by the hardware. ... of a crc32 of the 4-tuple in cxgb. ... of this is moving the queues into the driver. ...
    (freebsd-current)
  • Re: TX Multiqueue?
    ... The rx hardware queue is determined by the hardware. ... of a crc32 of the 4-tuple in cxgb. ... of this is moving the queues into the driver. ...
    (freebsd-net)
  • RE: TX Multiqueue?
    ... The rx hardware queue is determined by the hardware. ... of this is moving the queues into the driver. ... Based upon our experience, form other OS, the HW TX/RX queues are ... now supported by NIC vendors. ...
    (freebsd-current)