Re: MAssive Performance hit after AIX 4.3 -> 5.2 upgrade!!

In article <1136513666.504567.291170@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>, christiancolborn <christiancolborn@xxxxxxxxx> wrote:
> the problem is constant, slow around the clock, and of course worse
> during peak usertime, but even when i ssh into the thing at night its
> ridiculously slow even to type commands!

Hmm, that really does sound like the system is memory and/or I/O-starved
(e.g. due to excessive paging *OR* due to application pounding the disk
so much or so hard).

My current hypothesis is that you were already low on memory in AIX
4.3.3 but borderline, and then may have gotten 'pushed over the limit'
with the upgrade to 5.2. Won't know if that's the case until can see
ps -efl and svmon -P output, though.

*IF* this is indeed related to memory, you should be able to buy
inexpensive but good quality Kingston memory.

They're MUCH cheaper than IBM; IBM really stiffs you with memory and
hard drive costs. Lots of AIX admins uses Kingston memory if they don't
have the budget for IBM's memory. http://www.kingston.com/

There's also eBay, too.

I would say 512 MB is probably a comfortable baseline for AIX systems.

I don't remember; did you say what type/model this server was?

If you do not know, you can see it next to the serial number on the
front of the machine. Format is:

xxxx-yyy
bb-ccccc

xxxx = type (7025, 7046, 7026, 9076, etc)
yyy = model (H70, B50, F80, 6H1, etc)

bb = serial number, country code part
ccccc = rest of serial number, unique number for a given country code

Or you can get it from the machine:

# lscfg -vp | grep Model: | grep IBM

It will probably be the first one listed. Will be "IBM,xxxx-yyy" for the
type and model.

You will need to know the t/m to order the correct RAM.

> also... perf.tools is not installed, and the box is like 200 miles
> away... is there any way i could get the proper toolsets installed
> without physical access to the machine?

Sure. It's a piece of cake, especially if you have a copy of the AIX 5.2
CDs where you are.

If you do, just mount it on a Windows or Linux system (or whatever you
have handy). They're just ISO9660 CDs, nothing fancy.

Once CD #1 is mounted, go this directory on the CD:

usr/sys/inst.images

Then upload (ftp, scp, whatever) the file 'bos.perf' to your remote box.
It's about 2.5 MB.

Put it in /tmp on the remote AIX box.

Unmount CD #1, mount CD #2, go to usr/sys/inst.images.

Upload the file 'perfagent.tools' to the remote box. It's about 1.8 MB.

Put it in /tmp on the remote AIX box.

Unmount CD #2 and put the CDs away.

Hop on the remote AIX box.

Do these commands to install it:

# inutoc /tmp
# installp -aXgd/tmp bos.perf perfagent.tools

I would also strongly recommend also applying AIX 5.2 patches if you
haven't already done so. Especially for stuff that touches deep in the
kernel like the performance monitoring tools.

If you want to free up some space after installing the packages, just do:

# rm /tmp/.toc /tmp/bos.perf /tmp/perfagent.tools

If you do not have the AIX 5.2 CDs locally, then I would strongly
suggest you have a copy of the CD set sent to you from one of your
sites. It makes it much easier to deal with things like this remotely.

I can describe how to do patching in a separate message if you're unsure
how to do it. It's pretty easy. Hard part is waiting to download maybe
600 MB of patches. :)

> i'm not sure about the actual app(s) running on the machine, so i cant
> grep for it (theres tons of stuff on ps -ef), but i can find out
> tommorrow.

Ok. Sounds good. And make sure you do ps -efl once you figure out which
app it is, 'cause that gives more details -- especially on memory usage,
that ps -ef doesn't show.

If you're just not sure what app, just post the whole output of ps -efl
without the grep.

It'll probably be easy to tell what the major app(s) are, based on
process's command line arguments , CPU usage, or memory usage.

> also... can i do that time dd command on other resources, like mem0 to
> test I/O times to individual pieces of memory? that would be cool.

I'm afraid that dd is only really useful for disk or tape based stuff.

I use dd as a quick rough-idea benchmarking tool to see if there's a
problem or not. If it looks solid, I don't usually whip out the really
fancy disk benchmarking tools like iozone or bonnie. (They're complex,
and takes much longer to measure.)

However, there are other add-on tools to exercise other resources, I
believe?

For instance, with my IBM tape drives, I've got a tool called 'tapeutil'
which has a built-in benchmarking mode.

> all your thoughts/ideas are greatly appreciated. i'm new to aix and
> it seems there are very few aix ninjas around. again, thanks.

Not a problem. Just keep asking questions 'til you understand or have an
answer. :)

Oh, I have another suggestion. If you can, post the output of 'errpt'.
It will print one-line summaries of every single logged error.

Sometimes, you can see a clear pattern of problems that leads up to a
significant event.

(To review each entry in detail, do 'errpt -a|more'. To just see the
list, 'errpt'.)

-Dan
.