Re: BSD VM architecture...
phil-news-nospam_at_ipal.net
Date: 06/05/03
- Next message: lredmond: "Reading Excel Files"
- Previous message: phil-news-nospam_at_ipal.net: "Re: fts(3) depth or breadth first?"
- In reply to: Steve H: "BSD VM architecture..."
- Next in thread: Steve H: "Re: BSD VM architecture..."
- Reply: Steve H: "Re: BSD VM architecture..."
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]
Date: 4 Jun 2003 22:51:34 GMT
On 4 Jun 2003 11:05:42 -0700 Steve H <netspam@shic.co.uk> wrote:
| I've been wondering about a quirky approach to improve the
| performance/capacity ratio for a VM hungry application I'm developing.
| Put simply, I've huge, but sparsely accessed files (which I mmap into
| my process' address space) and a few megs of compact and very
| frequently accessed control structures. When considering going
| multi-user (one set of control structures and many independent files
| to mmap), I'm faced with two options:
|
| * Use 1 process per user (hence each user has lots of VM space
| available without affecting other users) and use inter-process
| communication to access the control structures (which would work, and
| is not restrictive on available VM space as number of concurrent users
| increases, but is pretty slow as a result of heavy inter-process
| communication.)
This solution is more secure. Even if you think you totally trust
every user, I cannot imagine a multi-user system without having some
level of security barriers involved or considered.
| * Use a single process with multiple threads and statically apportion
| VM space between a fixed number of users. This allows more rapid
| access of the control structures, but at the cost of reduced VM space
| available to each user, but seriously restricts scalability.
Unless you get a good 64-bit architecture going. Ultrasparc, Itanium,
Opteron, and the coming PPC-64, are possibilities.
| What I'm considering is a mechanism to allow a single process (maybe
| with multiple threads) to switch between accessing multiple "VM
| spaces" using some custom kernel extension. This idea rests on a few
| assumptions:
You want to switch the various "subspaces" in and out of the VM? You
can do that now. The system call is mmap(). Just make each space be
a file. Leave them open if you have enough file descriptors. You can
also map in just portions of each subspace as needed. I have used this
technique to map files larger than 4GB by calling mmap() on on the
parts I need at a given time. Though I did not, had I run on a system
with a memory model that allowed, and had, more than 4GB of RAM, then
I could have had a file larger than 4GB entirely in RAM, and view it
with a process limited to a maximum VM of 4GB, in chunks smaller than
that (perhaps one 512MB chunk at a time, or perhaps several 1MB chunks).
| 1. It should be relatively simple to extend the VM architecture to
| allow these multiple contexts.
See above. Negotiating space via mmap() might work for you.
| 2. There must be a way around the apparent problem of retaining access
| to the VM space into which the current process' executable data is
| mapped and, of course, retaining the execution stack.
In Linux the usual VM setup you have about 1GB for memory mapping. I don't
know the BSD situation. It's just a matter of using it wisely.
| 3. It must be significantly quicker to switch "VM spaces" than it is
| to use conventional inter-process communication.
| As I find it hard to believe I'm the first person to ponder about this
| idea, I'd like to enquire if anyone knows of any previous documented
| attempts to offer this sort of facility in BSD (or any other Unix-like
| OS, for that mater)
|
| Ideas anyone?
You could also have some portion of address space shared between all
processes for the bulk communication, then use the IPC to send short
messages like "Go access the 1MB buffer at 0x12300000 for operation FOO".
If the amount that needs to be shared to effectively communicate is not
larger than a significant portion of each VM, this should work. If the
work processes are parented from a common process, this is easy to do.
Or you can designate a file to be shared, and open() and mmap() it in
each process. Or if the file needs to be more secure, open() and mmap()
it in a common parent. Descriptor transferring might be another way to
get the file into the hands of each work process w/o compromising the
permissions of the file.
-- ----------------------------------------------------------------- | Phil Howard - KA9WGN | Dallas | http://linuxhomepage.com/ | | phil-nospam@ipal.net | Texas, USA | http://ka9wgn.ham.org/ | -----------------------------------------------------------------
- Next message: lredmond: "Reading Excel Files"
- Previous message: phil-news-nospam_at_ipal.net: "Re: fts(3) depth or breadth first?"
- In reply to: Steve H: "BSD VM architecture..."
- Next in thread: Steve H: "Re: BSD VM architecture..."
- Reply: Steve H: "Re: BSD VM architecture..."
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]
Relevant Pages
|
