Re: Binary data in multiple adjacent files
- From: "Richard B. Gilbert" <rgilbert88@xxxxxxxxxxx>
- Date: Thu, 26 Oct 2006 08:36:18 -0400
JF Mezei wrote:
I have a program that was setup to provide elevation data for all of
australia new zealand from 2 large datasets that covered that territory.
The application kpet the 2 files opened and did relative reads into
those binary files to obtain the required data (and cached those record
so that if the next request was for a nearby cell, it would save an IO).
Now however, updated data comes in much smaller files, each covering
only 1° lat by 1° long. Each file has 1201 * 1201 values, each value is
2 bytes. (basically elevation in metres for a 92 square metre area).
Each file is roughly 2.8 megs.
(This is degraded information due to the US government feeling it could
be used by terrorists, but it is still better than what was available
before. The SRTM data is precise to about 1 square metre, and some
sections have been released to 30 square metre accuracy, but most is at
92 metre).
Switching to this would mean that my application would need to open
hundreds of files to cover australia for instance, so this introduces
scalability issues. And since I need to review the code, I figured I
should perhaps consider other methods.
For my current needs, I need to cover from 45 to 47° north, and from
-73° to -76°, which gives me th following files:
N46W076 N46W075 N46W074
N45W076 N45W075 N46W074
N46W074 covers 46°N 74°W at the lower left corner to 47°N 73°W at the
upper right corner.
Looking at $CRMPSC, it appears I need to do a RMS $OPEN on each file and
provide the channel associated with each file.
So this would eat into the FILLM quota.
Now, if i just do the $CRMPSC to map a whole file to some virtual
address in my process space, would this consume 2,8 meg of working set
right away ? Or does VMS just allocate virual memory that is marked
invalid, and only when I try to access a few bytes at a location would
VMS load a single page from the actual file that contain those 2 bytes I
asked for ?
Virtual memory is just that! Things only need to occupy physical memory when you try to read or write them. Anything you are not using can be pushed out to make room for something else. So it won't necessarily eat your working set which is physical memory. There are also limits on virtual memory (PAGEFILEQUO) and a real constraint. The later is that performance will suck if if each memory reference is to a new page and you don't have a large working set.
Not that I know of. Virtual memory is managed globally for your process. When you need a new page (you requested a page not in your working set) an old page gets pushed out to make room. The least recently used page will normally be the one pushed out.
The nature of my application means that as I follow a route, I read data
progressing in one direction, and the odds of having to go back are low
(but not nil). So, blocks containing elevation data read early on are unlikely to be needed again once I have moved on to points outside those blocks.
Does VMS provide a means where I can specify that I want at max 2 pages
from a global section to mapped to my working set at ay point in time ?
(this way, when I request a shortword located in a different block, the
system would automatically unmap the oldest block mapped to that file
and use that memory to map the new block to the file).
<snip>
.
- References:
- Binary data in multiple adjacent files
- From: JF Mezei
- Binary data in multiple adjacent files
- Prev by Date: Re: DS10L surgery report
- Next by Date: Re: Alternate file types for RUN ?
- Previous by thread: Re: Binary data in multiple adjacent files
- Next by thread: Re: Binary data in multiple adjacent files
- Index(es):
Relevant Pages
|
