Re: Changes in the network interface queueing handoff model


Robert Watson writes:

5BOne of the ideas that I, Scott Long, and a few others have been bouncing
around for some time is a restructuring of the network interface packet
transmission API to reduce the number of locking operations and allow network
device drivers increased control of the queueing behavior. Right now, it

<....>

- The ifnet send queue is a separately locked object from the device driver,
meaning that for a single enqueue/dequeue pair, we pay an extra four lock
operations (two for insert, two for remove) per packet.


Going forward, especially now that we support sun4v CoolThreads
hardware, we're going to want to rethink the "single lock" per
transmit routine model that most drivers have. The most expensive
operation in transmit routines is bus_dmamap_load_mbuf_sg(),
especially when there is an IOMMU involved (like on CoolThreads
machines) and there is no reason why this needs to be called with a
driver's transmit lock held. I have hard data (from Solaris) about
how much fine grained locking in a 10GbE driver's transmit routine
helps.

Drew

_______________________________________________
freebsd-arch@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: Changes in the network interface queueing handoff model
    ... meaning that for a single enqueue/dequeue pair, we pay an extra four lock ... especially now that we support sun4v CoolThreads ... how much fine grained locking in a 10GbE driver's transmit routine ...
    (freebsd-net)
  • Re: Changes in the network interface queueing handoff model
    ... > meaning that for a single enqueue/dequeue pair, we pay an extra four lock ... Going forward, especially now that we support sun4v CoolThreads hardware, we're going to want to rethink the "single lock" per transmit routine model that most drivers have. ... Right now, with the exception of locking for the ifnet dispatch queue, I believe our ifnet API pretty much leaves decisions about the nature and granularity of synchronization to the device driver author. ...
    (freebsd-net)
  • Re: Changes in the network interface queueing handoff model
    ... > meaning that for a single enqueue/dequeue pair, we pay an extra four lock ... Going forward, especially now that we support sun4v CoolThreads hardware, we're going to want to rethink the "single lock" per transmit routine model that most drivers have. ... Right now, with the exception of locking for the ifnet dispatch queue, I believe our ifnet API pretty much leaves decisions about the nature and granularity of synchronization to the device driver author. ...
    (freebsd-arch)
  • Re: Changes in the network interface queueing handoff model
    ... meaning that for a single enqueue/dequeue pair, we pay an extra four lock ... we're going to want to rethink the "single lock" per transmit routine model ... granularity of synchronization to the device driver author. ...
    (freebsd-net)
  • Re: Changes in the network interface queueing handoff model
    ... meaning that for a single enqueue/dequeue pair, we pay an extra four lock ... we're going to want to rethink the "single lock" per transmit routine model ... granularity of synchronization to the device driver author. ...
    (freebsd-arch)