Interesting cross-fertilization with DfBSD
From: Andre Oppermann (andre_at_freebsd.org)
Date: 03/31/04
- Previous message: Ken Smith: "Testing a patch to syscons(4)"
- Next in thread: Robert Watson: "Re: Interesting cross-fertilization with DfBSD"
- Reply: Robert Watson: "Re: Interesting cross-fertilization with DfBSD"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Wed, 31 Mar 2004 20:23:52 +0200 To: current@freebsd.org
attached mail follows:
Date: Tue, 30 Mar 2004 14:58:13 -0800 (PST)
The recent PIPE work adapted from Alan Cox's work in FreeBSD-5 has really
lit a fire under my seat. It's amazing how such a simple concept can
change the world as we know it :-)
Originally the writer side of the PIPE code was mapping the supplied
user data into KVM and then signalling the reader side. The reader
side would then copy the data out of KVM.
The concept Alan codified is quite different: Instead of having the
originator map the data into KVM, simply supply an array of vm_page_t's
to the target and let the target map the data into KVM. In the case
of the PIPE code, Alan used the SF_BUF API (which was originally developed
by David Greenman for the sendfile() implementation) on the target side
to handle the KVA mappings.
Seems simple, eh? But Alan got an unexpectedly huge boost in performance
on IA32 when he did this. The performance boost turned out to be due
to two facts:
* Avoiding the KVM mappings and the related kernel_object manipulations
required for those mappings saves a lot of cpu cycles when all you
want is a quick mapping into KVM.
* On SMP, KVM mappings generated IPIs to all cpus in order to
invalidate the TLB. By avoiding KVM mappings all of those IPIs
go away.
* When the target maps the page, it can often get away with doing
a simple localized cpu_invlpg(). Most targets will NEVER HAVE TO
SEND IPIs TO OTHER CPUS. The current SF_BUF implementation still
does send IPIs in the uncached case, but I had an idea to fix that
and Alan agrees that it is sound... and that is to store a cpumask
in the sf_buf so a user of the sf_buf only invalidates the cached
KVM mapping if it had not yet been accessed on that particular cpu.
* For PIPEs, the fact that SF_BUF's cached their KVM mappings
reduced the mapping overhead almost to zero.
Now when I heard about this huge performance increase I of course
immediately decided that DragonFly needed this feature to, and so we
now have it for DFly pipes.
Light Bulb goes off in head
But it also got me to thinking about a number of other sticky issues
that we face, especially in our desire to thread major subsystems (such
as Jeff's threading of the network stack and my desire to thread VFS),
and also issues related to how to efficiently pass data between threads,
and how to efficiently pass data down through the I/O subsystem.
Until now, I think everyone here and in FreeBSD land were stuck on the
concept of the originator mapping the data into KVM instead of the
target for most things. But Alan's work has changed all that.
This idea of using SF_BUF's and making the target responsible for mapping
the data has changed everything. Consider what this can be used for:
* For threaded VFS we can change the UIO API to a new API (I'll call it
XIO) which passes an array of vm_page_t's instead of a user process
pointer and userspace buffer pointer.
So 'XIO' would basically be our implementation of target-side mappings
with SF_BUF capabilities.
* We can do away with KVM mappings in the buffer cache for the most
prevalient buffers we cache... those representing file data blocks.
We still need them for meta-data, and a few other circumstances, but
the KVM load on the system from buffer cache would drop by like 90%.
* We can use the new XIO interface for all block data referencse from
userland and get rid of the whole UIO_USERSPACE / UIO_SYSSPACE mess.
(I'm gunning to get rid of UIO entirely, in fact).
* We can use the new XIO interface for the entire I/O path all the way
down to busdma, yet still retain the option to map the data if/when
we need to. I never liked the BIO code in FreeBSD-5, this new XIO
concept is far superior and will solve the problem neatly in DragonFly.
* We can eventually use XIO and SF_BUF's to codify copy-on-write at
the vm_page_t level and no longer stall memory modifications to I/O
buffers during I/O writes.
* I will be able to use XIO for our message passing IPC (our CAPS code),
making it much, much faster then it currently is. I may do that as
a second step to prove-out the first step (which is for me to create
the XIO API).
* Once we have vm_page_t copy-on-write we can recode zero-copy TCP
to use XIO, and won't be a hack any more.
* XIO fits perfectly into the eventual pie-in-the-sky goal of
implementing SSI/Clustering, because it means we can pass data
references (vm_page_t equivalents) between machines instead of
passing the data itself, and only actually copy the data across
on the final target. e.g. if on an SSI system you were to do
'cp file1 file2', and both file1 and file2 are on the same filesystem,
the actual *data* transfer might only occur on the machine housing
the physical filesystem and not on the machine doing the 'cp'. Not
one byte. Can you imagine how fast that would be?
And many other things. XIO is the nutcracker, and the nut is virtually
all the remaining big-ticket items we need to cover DragonFly.
This is very exciting to me.
-Matt
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
- Previous message: Ken Smith: "Testing a patch to syscons(4)"
- Next in thread: Robert Watson: "Re: Interesting cross-fertilization with DfBSD"
- Reply: Robert Watson: "Re: Interesting cross-fertilization with DfBSD"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
|
