Re: MAssive Performance hit after AIX 4.3 -> 5.2 upgrade!!

yes, i think AIX has a sort of pre-emptive paging system that makes it
always look sort of high...
the problem is constant, slow around the clock, and of course worse
during peak usertime, but even when i ssh into the thing at night its
ridiculously slow even to type commands!
also... perf.tools is not installed, and the box is like 200 miles
away... is there any way i could get the proper toolsets installed
without physical access to the machine?
i wanted to use some other things from that toolset too.

i'm not sure about the actual app(s) running on the machine, so i cant
grep for it (theres tons of stuff on ps -ef), but i can find out
tommorrow.

also... can i do that time dd command on other resources, like mem0 to
test I/O times to individual pieces of memory? that would be cool.

all your thoughts/ideas are greatly appreciated. i'm new to aix and
it seems there are very few aix ninjas around. again, thanks.





Dan Foster wrote:
> In article <1136505541.650991.111340@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>, christiancolborn <christiancolborn@xxxxxxxxx> wrote:
> > Dan the Man!
>
> :)
>
> > # lsps -a
> > Page Space Physical Volume Volume Group Size %Used Active
> > Auto Type
> > hd6 hdisk1 rootvg 256MB 41 yes
> > yes lv
>
> 41% used? Hmm. That suggests you might be seeing paging activity which
> is a big performance killer if it has to go so far into paging space.
>
> I'm not sure because AIX may reserve paging space, but not actually
> start to page in/out unless it is running low on memory.
>
> It's not yet clear exact culprit or why. Need to investigate further
> with svmon.
>
> > # svmon -G
> > ksh: svmon: not found.
>
> This is a must-have tool, especially for debugging memory related
> issues.
>
> To install it, you will need to install perfagent.tools and
> bos.perf.tools.
>
> If you already have these installed, it should be /usr/sbin/svmon.
>
> Once you have it, look at 'svmon' output and also 'svmon -U|more'
> output.
>
> You can post 'svmon' output here because it is short.
>
> svmon -P has a lot of information, but it tells you breakdown of memory
> usage including what is using how much of paging space.
>
> It will probably be too large to post 'svmon -P' output here. You could
> make it temporarily available on a web site, e-mail the output to me, or
> analyze the results yourself.
>
> svmon -P is great because it lets you figure out if you should blame the
> OS or blame the application for excessive memory usage. :)
>
> > # netstat -v en0|grep Media
> > Media Speed Selected: 100 Mbps Full Duplex
> > Media Speed Running: 100 Mbps Full Duplex
>
> Confirmed that network connection appears to be set up and seen ok.
>
> > # time dd if=/dev/rhdisk0 of=/dev/null bs=64k count=10000
>
> The dd tests shows 25-30 MB/sec for all disks, which is healthy.
>
> Also wanted to rule out possibility of a failing disk which can cause a
> performance hit due to SCSI bad block sparing if disk is on way out in
> certain circumstances. But does not appear to be a problem here.
>
> > # netstat -i 1
> > input (en0) output input (Total) output
> > packets errs packets errs colls packets errs packets errs colls
> > 3328612 0 3720965 0 0 3586536 0 3979085 0 0
> > 11 0 10 0 0 11 0 10 0 0
> > 35 0 36 0 0 37 0 38 0 0
> > 18 0 16 0 0 18 0 16 0 0
> > 30 0 21 0 0 30 0 21 0 0
>
> Very quiet network activity, and no errors or collisions. Very good.
>
> Also still need to know if the problem is constant or occasional?
>
> What application does the machine run?
>
> Can you post output of ps -efl | grep <application> ?
>
> That would offer insight into memory sizing by the application.
>
> It also helps figure out if its memory use is excessive (for example,
> due to an application bug like a memory leak).
>
> -Dan (the Man)

.