Re: RFC: FreeBSD I/OAT driver

Andrew Gallatin [gallatin@xxxxxxxxxxx] wrote:

Jack Vogel writes:
> We are making our development driver for the I/OAT engine available for
> download, experimentation, and comment available at:
>
> http://sourceforge.net/project/showfiles.php?group_id=42302&package_id=202220
>
> This includes a core driver for the dma hardware and a set of stack changes
> to allow use of the engine on the receive side of the stack.
>
> There are certainly rough edges and limitations in this code, but we have run
> it internally and seen some great results.
>
> I would like to see this get into CURRENT, so anything Prafulla and I can do
> to help or answer questions, send us email.

Excellent! Can you share some of these results? I would love to try
it, but I don't have FreeBSD on any machine with I/OAT hardware.


Dual core Woodcrest
Bensley Chipset
Netperf Receive Test - 64k IO size
6.1-RELEASE SMP Kernel
MTU 1500 bytes

Num Ports Thr(Native) Thr(I/OAT) CPU (Native) CPU (I/OAT)
(Mbps) (Mbps) (%) (%)
1 943 943 14 11
2 1886 1886 46 22
4 1945 2531 84 54


It scales fairly linearly as number of ports increase.
Haven't run more than 4 port test with FreeBSD though.


I've taken a very quick look at it. Maybe I'm just being dense,
but I don't like the name "dma_" being in the global namespace.
Maybe things (like dma_*_list should be called at least
dmaengine_*_list, etc.

Yeah - probably ioatdma_ would be more clear.


There are some style(9) defects which I'm sure others who are more
proficient at style(9) than I am will point out (// comments, function
names not starting in column 0, etc).

How deep would you expect so->dma_wait_queue to get? Would it make
sense to keep a pointer to the last item so that insertion is O(1),
rather than O(N)?

The size is dependant on Application IO size and MSS.
So you are right it might be useful to keep a pointer to last item.
At higher IO sizes it can get quite deep for 1500 byte MTU.


Would it be possible to have a sysctl tunable threshold, below which
the system does a normal uiomove? A normal copyout() will certainly
be faster at some point..


Yes - it doesn't make sense to use the DMA engine for small packets.
Also you get more benefit if you overlap IO with computation.

Thanks for the great work!

Thank you for your help earlier.

_______________________________________________
freebsd-net@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@xxxxxxxxxxx"