Re: WORKING_SET.COM gives all COM states when run at high priority

From: Jeff Chimene (jchimene_at_earthlink.net)
Date: 07/06/05 

Date: Wed, 06 Jul 2005 17:29:31 GMT

AEF wrote:
> Jeff Chimene wrote:
>
>>AEF wrote:
>>
>>>Jeff Chimene wrote:
>>>
>>>
>>>>AEF wrote:
>>>>
>>>>
>>>>>Jeff Chimene wrote:
>>>>>
>>>>>
>>>>>
>>>>>>AEF wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>Larry Kilgallen wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>In article <1120138193.202168.136920@o13g2000cwo.googlegroups.com>, "AEF" <spamsink2001@yahoo.com> writes:
>>>>>>>>
>
> [...]
>
>>>>Nevertheless, I feel compelled to ask: is this trading system I/O or
>>>>compute bound? It sounds like you've decided it's compute bound. In that
>>>
>>>
>>>Network-I/O bound. The trading system startup get things very busy both
>>>CPU- and I/O-wise but that is done before the trading session begins so
>>>shaving a few minutes there doesn't really matter.
>>
>>Agreed. Unless your users are twiddling their thumbs waiting for a reboot...
>
>
> Users aren't even in their offices yet.
>
>
>>>>case, I'm predicting you'll settle on dividing physical memory into
>>>>blocks and setting WSDEF by task to allocate those blocks.
>>>
>>>I'm doing experiments to determine the elapsed time/CPU time/page
>>>faults vs. working set size by running the three relevant programs with
>>>various fixed working set sizes and looking up the stats with SHOW
>>>SYSTEM, SHOW PROC/CONT, ACCOUNTING, etc. It seems to me that you want
>>>*some* difference in page fault rate between running at wsquota vs.
>>>wsextent. Otherwise, what's the point of borrowing memory?
>>
>>Depending on some applications, I'm one of those who believe that while
>>VMS and VAX/Alpha were designed to page, it's not always a Good Thing.
>
>
> I'm not sure what you're saying here. If the free list is large enough,
> the process will grow to the point where it either reaches its WSEXTENT
> or to where the page fault rate drops below PFRATH. Now if the elapsed
> time for, say, compiling a PASCAL program is equal to 24 seconds for
> working set sizes ranging from 2048 to 6064, then what's the point of
> the last 4000 or so pages when the performance is the same? Those 4000
> pages could be used by other processes. So if all PASCAL codes act that
> way, couldn't one then argue for WSEXTENT of 2048 at least on a
> memory-limited system? Then I figure that WSQUOTA should give
> measurably worse results than WSEXTENT because otherwise what's the
> point of using more memory if it doesn't improve the performace? The
> manual itself says to set WSQUOTA so that one gets "reasonable"
> performance. And I like to make WSDEFAULT half of WSQUOTA or less.

Agreed. My point was w/r/t production applications running on a more or
less "stable" system. Throwing compilation into the mix creates fruit
salad. If one knows the memory requirements for these "server" apps, one
can adjust VM settings to allocate the entire WS at image activation
time. Coupled with the proper user-written image initialization, the
page fault rate could approach zero for the image duty cycle. The
interesting bits are where the database engine gets involved.

Are you using the production machine as a development environment? All
bets are off if that's the case.

>>>And if the
>>>curve is really flat for large working sets I'd think that setting
>>>WSEXTENT to a value which is somewhat lower than WSMAX (for processes
>>>running that program) would be a good idea. I get different curves for
>>>different programs. I was going to set WSDEF to half of WSQUOTA.
>>
>>I'm wondering how accurate your performance tests are. I don't see
>>MONITOR in your tool set listed above. Also, I've been out of the VMS
>>loop too long to know what HP would recommend for performance tuning. I
>>also recall a very nice simulation tool that implemented a BASIC-like
>>scripting language.
>
>
> I also use MONITOR when appropriate.

Good. Do you have MONITOR running in the background using that
DecHPAQ-supplied DCL script? Loading the logs into an analysis tool can
be a good way to proactively address performance bottlenecks.

>>>But then there's the proactive memory reclamation and that definitely
>>>makes things interesting! So far that has worked quite well on the
>>>systems I have that are already tight on memory. I may have some
>>>questions about that soon.
>>
>>Tight memory systems seem so counterproductive in a trading environment.
>>Is there no way to add memory to these systems? If not, how about a disk
>>dedicated to nothing but paging? If these systems are swapping, you are
>>already behind the eight ball.
>
>
> Looking at the display of MONITOR SYSTEM during trading hours shows
> very little activity. It's quite dull. A few dozen executions is a
> medium day for one desk and a good day on the other. And maybe a couple
> hundred prices posted. Each of these events takes a miniscule amount of
> CPU, PF's, I/O, etc. so is it even worth it for me to improve the WS
> settings I inherited?

Probably not. It sounds like this system is well tuned, and you'll have
to look elsewhere for bottlenecks.

> A few of our systems our tight on memory (free
> list near 2*freegoal and a small number of pages in the virutal I/O
> cache) and VMS appears to be trimming as necessary. Sometimes there is
> one swapped process slot (usually REMACP).

Seems like all's quiet on the Western Front...

>>>>You mention "doubling the number of data feed jobs ... make memory
>>>>tight" Are you using the mapped section system services?
>>>
>>>
>>>I don't know. I didn't write any of the compiled programs. I know that
>>>one of the programs locks a good portion of memory into physical
>>>memory. I know that shared memory is used. There are three such files
>>>that have to be installed shareable and writable. I don't know if that
>>>answers your question.
>>
>>Yes, it does. It sounds like those folks knew what they wanted in a
>>virtual memory environment. What tuning suggestions can they provide for
>>VMS?
>
>
> "They" are gone. This is a(n) -- AUGH -- legacy app. We have a
> developer who knows PASCAL if any bugs need to be fixed or features
> added, but he is very busy doing lots of other stuff and there are no
> pressing bugs for him to fix.

At this point, I'd ask the user community if response time meets their
(realistic) expectations. After adjusting the responses for outliers who
want sub-second response time on that infrequent I/O intensive database
lookup, you should discover whether or not further tuning is warranted.

> There's a long history of various things here that I'd rather not go
> into here. If you have further questions, please send email to
>
> spamsink2001 Mary had a little lamb
>
> atski It's fleece was white as snow.
>
> yahoo and everywhere that Mary went
>
> dotski
>
> com the lamb was sure to go.
>
> or just got to Google groups and use the Google "reply to author"
> button.

I like the e-addr obfuscation.

> I appreciate your help and interest; thanks.

You're welcome.

Cheers,
jec