Re: Performance problem
From: Patrick B. O'Brien (pobrien_at_DOIT.NV.GOV)
Date: 08/02/05
- Previous message: Fernandez Garay Jorge: "Performance problem"
- Maybe in reply to: Fernandez Garay Jorge: "Performance problem"
- Next in thread: Green, Simon: "Re: Performance problem"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Tue, 2 Aug 2005 14:10:34 -0700 To: aix-l@Princeton.EDU
IBM suggests you troubleshoot the bottle neck areas in the following
order; they say to find the CPU bound stuff first. Looks to me like your
almost CPU bound and your heavily paging.
Cpu bound?
Memory bound?
Disk bound?
Network bound?
I'd look at these 2 utils first;
Ps auxwww|more
Then maybe do a tprof -x sleep 30. The out file will be a __prof.all.
Look at your total column.
I'm sure others have better ideas.
-----Original Message-----
From: IBM AIX Discussion List [mailto:aix-l@Princeton.EDU] On Behalf Of
Fernandez Garay Jorge
Sent: Tuesday, August 02, 2005 1:33 PM
To: aix-l@Princeton.EDU
Subject: Performance problem
I wonder if someone in the list can help us determining the causes of
this problem.
Perhaps someone can, at least, direct us to look in some direction.
Any help will be appreciated
Thanks in advance
Jorge
We have an S80 box (AIX 4.3.3), loaded with intensive database
applications (Oracle 8i).
12 Processors
10 GB RAM
Data Base Datafiles located at a Massive Storage device (EVA 5000 from
HP). None raw devices,
(Normal JFS filesystems)
The Databases are configured to perform Asynchronous I/O
Some time ago, we have tunned VMM with following command
/usr/samples/kernel/vmtune -p5 -P20 -f720 -F768 -W256
It had been working fine, till past friday.
sar command output shows a heavy %wio (50-60%)
vmstat command output shows large values for wait queues (65-70)
We couldn't identify any additional workloads.
We have added 6 processors, and enlarged the 'maxserver' parameter for
Asynchronous I/O, from 120
to 180 ( 10 x Number of Processors)
New 'minserver' parameter value was enlarges from 60 to 90 (New
'maxserver' value/2)
After that, things are getting far from better.
Here is a sample of sar command output
sar 1 50
AIX sp17 3 4 000B85FD4C00 08/02/05
16:05:50 %usr %sys %wio %idle
16:05:51 18 7 58 17
16:05:52 24 10 48 18
16:05:53 22 10 50 18
16:05:54 26 9 49 16
16:05:55 34 9 45 12
16:05:56 33 11 46 10
16:05:57 33 9 47 11
16:05:59 40 10 42 9
16:06:00 43 10 41 6
16:06:01 52 12 32 4
16:06:02 51 9 36 4
16:06:03 50 11 35 5
16:06:04 47 8 40 5
16:06:05 38 7 49 6
16:06:06 54 11 32 3
16:06:07 59 8 31 3
16:06:08 49 7 41 2
16:06:09 46 6 45 3
16:06:11 55 8 35 2
16:06:12 53 9 36 1
16:06:13 62 11 25 2
16:06:14 64 11 22 2
16:06:15 67 8 23 3
16:06:16 68 10 21 1
16:06:17 60 7 31 2
16:06:18 48 9 37 6
16:06:19 64 11 23 2
16:06:20 61 10 25 3
16:06:21 56 9 31 4
16:06:23 57 9 30 3
16:06:24 61 11 24 3
16:06:25 58 11 27 4
16:06:26 48 9 39 5
16:06:27 36 10 41 13
16:06:28 43 10 38 8
16:06:29 37 9 44 10
16:06:30 32 6 50 12
16:06:31 40 8 41 11
16:06:32 34 7 47 12
16:06:33 31 7 50 13
16:06:35 28 7 40 25
16:06:36 30 6 43 20
16:06:37 40 5 39 15
16:06:38 33 8 40 18
16:06:39 33 9 45 13
16:06:40 32 7 45 15
16:06:41 36 11 41 12
16:06:42 31 13 40 16
16:06:43 29 6 48 18
16:06:44 36 7 41 17
Average 44 9 39 9
And a sample of vmstat command output
vmstat 1 50
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
6 18 2438992 809 0 10 24 4471 4506 0 4612 17675 13261 24 10 34
32
11 34 2439053 781 0 40 0 5203 35268 0 7660 50194 21233 45 18 6
31
9 33 2440977 789 0 12 137 4996 32334 0 5190 35689 16190 28 16 10
46
7 27 2440930 802 0 17 671 3655 20331 0 7060 50285 18872 44 19 9
28
12 23 2440614 762 0 33 0 4299 38153 1 6721 41695 17588 45 17 5
34
10 31 2439757 738 0 31 0 5549 50557 0 7141 56357 17151 54 16 4
26
16 25 2439858 749 0 30 0 7751 53101 0 8361 62060 18989 69 16 1
14
10 22 2441191 1232 0 24 93 6605 44223 0 7235 73798 14944 63 15 3
19
9 28 2442525 735 0 21 625 4450 28052 0 6008 46730 12464 50 7 6
37
11 24 2444507 593 0 24 1398 8628 58952 1 8593 61438 18229 57 12
5 27
9 25 2444581 785 0 19 927 4084 26364 0 6055 35261 13674 38 6 12
44
20 23 2444787 741 0 40 561 5183 45328 0 7788 57485 17235 56 11 5
28
17 22 2445163 879 0 85 508 5843 44913 1 8068 51444 16850 51 9 5
35
10 23 2444774 865 0 51 0 4826 34555 0 6730 36549 13996 46 8 5
41
10 31 2444832 1163 0 35 37 5342 38956 0 7298 48694 15752 45 8 6
41
11 28 2444949 807 0 25 0 6087 48390 0 8259 35130 19046 56 8 3
33
10 29 2445473 756 0 18 0 6702 42901 1 7709 40053 15677 47 7 5
40
6 30 2444814 763 0 20 1 6266 50960 0 7669 42844 15219 51 6 6
36
11 25 2444266 750 0 21 0 5670 48804 0 6239 43833 11396 46 7 8
39
11 26 2442489 1490 0 59 0 3691 30237 0 7199 36594 13563 46 7 6
41
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
7 27 2443850 742 0 50 0 4471 36921 0 6408 38074 12219 40 6 9
44
8 25 2444191 737 0 22 0 4203 33509 0 7514 30241 14809 32 6 16
46
30 23 2444757 901 0 14 0 4908 40912 0 6951 41559 13735 41 7 8
43
10 30 2441444 2877 0 25 0 5637 35870 0 7157 43121 14268 50 9 4
37
13 34 2441838 753 0 19 0 6302 38246 0 7748 36873 17299 41 9 5
45
8 35 2439540 742 0 70 0 6132 43070 0 7870 40044 17106 40 9 5
46
9 33 2440240 748 0 34 0 8254 74561 0 7014 41336 12791 46 8 5
42
10 33 2442476 760 0 41 0 8056 64271 0 7981 43419 16296 45 8 4
42
10 34 2443802 769 0 35 0 9184 74247 0 7510 41972 16446 39 10 6
46
9 26 2443227 770 0 5 0 8965 76191 0 7912 35435 16411 42 9 7
42
14 35 2443056 745 0 12 0 6795 70111 0 7398 36907 15777 45 9 7
40
11 26 2443738 742 0 17 0 8757 71219 0 6695 32048 14658 44 6 4
46
14 31 2442714 699 0 6 0 7571 64102 0 6857 30476 13955 47 8 3
42
10 28 2442842 783 0 18 0 6637 50274 0 7560 38011 14786 55 7 3
35
12 36 2441068 750 0 15 0 4970 35018 0 8199 50480 16553 47 10 4
39
9 34 2441826 744 0 9 0 8338 58961 0 8279 33540 16499 37 9 4
51
10 31 2442202 771 0 7 0 7690 47791 0 7871 33007 16183 36 9 7
47
7 34 2440987 765 0 9 0 5482 31760 0 7733 26424 14703 33 6 6
54
9 30 2441319 760 0 11 0 7728 53200 0 7732 40858 16125 39 9 6
46
12 29 2441580 764 0 13 0 10642 95595 0 7954 35019 16416 46 9
4 41
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
10 30 2442589 745 0 7 0 12540 118092 0 7934 46268 17159 42 10
5 43
11 28 2443661 761 0 13 0 10158 96434 0 7510 42608 16230 37 11
5 47
6 43 2439781 744 0 26 0 5845 43958 0 7875 51307 16445 37 10 6
47
7 38 2439617 717 0 24 0 9300 73314 0 8664 60575 20777 37 12 6
45
8 44 2439926 742 0 83 0 10806 83304 0 8304 53098 19277 35 14
5 47
5 42 2440404 776 0 80 0 8721 72482 0 8264 54994 20555 41 12 5
42
5 42 2440416 732 0 33 0 9635 79317 0 8150 49272 18550 45 10 3
42
15 35 2439968 718 0 17 0 8258 51476 0 8206 42273 17451 40 10 3
47
10 33 2437272 832 0 0 0 0 0 0 0 0 0 33 15 4 48
10 39 2436790 816 0 18 3 10630 68739 0 9357 42627 21451 32 15
5 48
Here is a sample of ps -f -k, to display kprocs raised as aioservers,
and their consumption:
aioservers are those having about 14:30 value in TIME column
ps -f -k
UID PID PPID C STIME TTY TIME CMD
root 0 0 120 Jul 31 - 13:11 swapper
root 516 0 120 Jul 31 - 2480:16 kproc
root 774 0 120 Jul 31 - 2445:24 kproc
root 1032 0 120 Jul 31 - 2452:55 kproc
root 1290 0 120 Jul 31 - 2471:57 kproc
root 1548 0 120 Jul 31 - 2467:07 kproc
root 1806 0 120 Jul 31 - 2461:22 kproc
root 2064 0 120 Jul 31 - 2505:23 kproc
root 2322 0 120 Jul 31 - 2519:31 kproc
root 2580 0 120 Jul 31 - 2495:13 kproc
root 2838 0 120 Jul 31 - 2492:40 kproc
root 3096 0 120 Jul 31 - 2509:50 kproc
root 3354 0 120 Jul 31 - 2511:25 kproc
root 3612 0 120 Jul 31 - 2543:46 kproc
root 3870 0 120 Jul 31 - 2546:04 kproc
root 4128 0 120 Jul 31 - 2553:04 kproc
root 4386 0 120 Jul 31 - 2532:15 kproc
root 4644 0 120 Jul 31 - 2551:52 kproc
root 4902 0 120 Jul 31 - 2566:15 kproc
root 5160 0 42 Jul 31 - 338:25 kproc
root 5418 0 0 Jul 31 - 0:05 kproc
root 5676 0 5 Jul 31 - 44:34 kproc
root 5934 0 0 Jul 31 - 0:09 kproc
root 6726 0 0 Jul 31 - 0:00 kproc
root 8040 1 0 Jul 31 - 14:27 kproc
root 8528 1 0 Jul 31 - 14:26 kproc
root 8812 1 0 Jul 31 - 14:14 kproc
root 9034 1 0 Jul 31 - 14:38 kproc
root 9292 1 0 Jul 31 - 13:59 kproc
root 9550 1 0 Jul 31 - 14:28 kproc
root 9808 1 0 Jul 31 - 0:00 kproc
root 10324 1 0 Jul 31 - 14:15 kproc
root 10582 1 1 Jul 31 - 14:16 kproc
root 10840 1 0 Jul 31 - 14:25 kproc
root 11098 1 0 Jul 31 - 14:06 kproc
root 11364 1 0 Jul 31 - 14:22 kproc
root 11612 1 0 Jul 31 - 0:00 kproc
root 11878 1 0 Jul 31 - 14:17 kproc
root 12134 1 0 Jul 31 - 14:27 kproc
root 12386 1 0 Jul 31 - 14:27 kproc
root 12642 1 0 Jul 31 - 14:32 kproc
root 12900 1 0 Jul 31 - 14:32 kproc
root 13158 1 0 Jul 31 - 14:24 kproc
root 13416 1 0 Jul 31 - 14:16 kproc
root 13674 1 3 Jul 31 - 14:08 kproc
root 13932 1 0 Jul 31 - 14:31 kproc
root 14190 1 0 Jul 31 - 14:16 kproc
root 14448 1 0 Jul 31 - 14:11 kproc
root 14706 1 0 Jul 31 - 14:30 kproc
root 14964 1 0 Jul 31 - 14:40 kproc
root 15222 1 0 Jul 31 - 14:20 kproc
root 15480 1 0 Jul 31 - 14:18 kproc
root 15738 1 0 Jul 31 - 14:25 kproc
root 15996 1 0 Jul 31 - 14:27 kproc
root 16254 1 0 Jul 31 - 14:08 kproc
root 16512 1 0 Jul 31 - 13:57 kproc
root 16770 1 0 Jul 31 - 14:13 kproc
root 17028 1 0 Jul 31 - 14:26 kproc
root 17286 1 0 Jul 31 - 14:33 kproc
root 17544 1 0 Jul 31 - 14:15 kproc
root 17802 1 0 Jul 31 - 14:10 kproc
root 18060 1 0 Jul 31 - 14:29 kproc
root 18318 1 0 Jul 31 - 14:30 kproc
root 18576 1 0 Jul 31 - 14:23 kproc
root 18834 1 0 Jul 31 - 14:22 kproc
root 19092 1 0 Jul 31 - 14:10 kproc
root 19350 1 0 Jul 31 - 14:08 kproc
root 19608 1 0 Jul 31 - 14:22 kproc
root 19866 1 0 Jul 31 - 14:13 kproc
root 20124 1 0 Jul 31 - 14:31 kproc
root 20382 1 0 Jul 31 - 14:17 kproc
root 20640 1 0 Jul 31 - 14:36 kproc
root 20898 1 0 Jul 31 - 14:33 kproc
root 21156 1 0 Jul 31 - 14:12 kproc
root 21414 1 0 Jul 31 - 14:27 kproc
root 21672 1 0 Jul 31 - 14:19 kproc
root 21930 1 0 Jul 31 - 14:28 kproc
root 22188 1 0 Jul 31 - 14:14 kproc
root 22446 1 0 Jul 31 - 14:22 kproc
root 22704 1 0 Jul 31 - 14:27 kproc
root 22962 1 0 Jul 31 - 14:19 kproc
root 23220 1 3 Jul 31 - 14:29 kproc
root 23478 1 0 Jul 31 - 14:19 kproc
root 23736 1 0 Jul 31 - 14:27 kproc
root 23994 1 0 Jul 31 - 14:28 kproc
root 24252 1 0 Jul 31 - 14:18 kproc
root 24510 1 0 Jul 31 - 14:30 kproc
root 24768 1 1 Jul 31 - 14:31 kproc
root 25026 1 0 Jul 31 - 14:27 kproc
root 25284 1 0 Jul 31 - 14:28 kproc
root 25542 1 0 Jul 31 - 14:08 kproc
root 25800 1 0 Jul 31 - 14:13 kproc
root 26058 1 0 Jul 31 - 14:14 kproc
root 26316 1 0 Jul 31 - 14:16 kproc
root 26574 1 0 Jul 31 - 14:09 kproc
root 26832 1 0 Jul 31 - 14:20 kproc
root 27090 1 0 Jul 31 - 14:23 kproc
root 27348 1 0 Jul 31 - 14:10 kproc
root 27606 1 0 Jul 31 - 14:13 kproc
root 27864 1 0 Jul 31 - 14:14 kproc
root 28122 1 0 Jul 31 - 14:24 kproc
root 28380 1 0 Jul 31 - 14:34 kproc
root 28638 1 0 Jul 31 - 14:29 kproc
root 28896 1 0 Jul 31 - 14:28 kproc
root 29154 1 0 Jul 31 - 14:12 kproc
root 29412 1 0 Jul 31 - 14:10 kproc
root 29670 1 0 Jul 31 - 14:14 kproc
root 29928 1 0 Jul 31 - 14:01 kproc
root 30186 1 1 Jul 31 - 14:34 kproc
root 30444 1 0 Jul 31 - 14:31 kproc
root 30702 1 0 Jul 31 - 14:14 kproc
root 30960 1 0 Jul 31 - 14:26 kproc
root 31218 1 0 Jul 31 - 14:24 kproc
root 31476 1 0 Jul 31 - 14:31 kproc
root 31734 1 0 Jul 31 - 14:20 kproc
root 31992 1 0 Jul 31 - 14:30 kproc
root 34862 0 0 Jul 31 - 0:00 kproc
root 35120 0 0 Jul 31 - 0:00 kproc
root 56012 1 0 Jul 31 - 14:11 kproc
root 56270 1 0 Jul 31 - 14:21 kproc
root 56528 1 0 Jul 31 - 14:05 kproc
root 56802 1 0 Jul 31 - 14:14 kproc
root 65644 1 0 Jul 31 - 14:10 kproc
root 66146 1 0 Jul 31 - 14:12 kproc
root 68018 1 0 Jul 31 - 14:25 kproc
root 68518 1 0 Jul 31 - 14:23 kproc
root 68938 1 3 Jul 31 - 14:17 kproc
root 69256 1 0 Jul 31 - 14:19 kproc
root 69552 1 0 Jul 31 - 14:26 kproc
root 70758 1 0 Jul 31 - 14:32 kproc
root 71996 1 0 Jul 31 - 14:28 kproc
root 72276 1 0 Jul 31 - 14:24 kproc
root 72598 1 0 Jul 31 - 14:19 kproc
root 74098 1 0 Jul 31 - 14:30 kproc
root 74920 1 2 Jul 31 - 14:13 kproc
root 76146 1 0 Jul 31 - 14:23 kproc
root 78936 1 0 Jul 31 - 14:34 kproc
root 79820 1 0 Jul 31 - 14:12 kproc
root 80658 1 0 Jul 31 - 14:16 kproc
root 81028 1 1 Jul 31 - 14:31 kproc
root 81792 1 0 Jul 31 - 14:19 kproc
root 82390 1 0 Jul 31 - 14:14 kproc
root 83348 1 0 Jul 31 - 14:12 kproc
root 83920 1 1 Jul 31 - 14:10 kproc
root 84706 1 0 Jul 31 - 14:14 kproc
root 86508 1 2 Jul 31 - 14:07 kproc
root 87920 1 0 Jul 31 - 14:21 kproc
root 88310 1 0 Jul 31 - 14:24 kproc
root 88532 1 0 Jul 31 - 14:16 kproc
root 89106 1 3 Jul 31 - 14:18 kproc
root 89842 1 0 Jul 31 - 14:23 kproc
root 90308 1 0 Jul 31 - 14:10 kproc
root 90566 1 0 Jul 31 - 13:59 kproc
root 91942 1 0 Jul 31 - 14:08 kproc
root 92394 1 0 Jul 31 - 14:08 kproc
root 92496 1 1 Jul 31 - 14:06 kproc
root 92722 1 0 Jul 31 - 14:18 kproc
root 93582 1 0 Jul 31 - 14:14 kproc
root 94806 1 0 Jul 31 - 14:32 kproc
root 95538 1 0 Jul 31 - 14:17 kproc
root 95820 1 2 Jul 31 - 14:13 kproc
root 96400 1 3 Jul 31 - 14:22 kproc
root 97270 1 0 Jul 31 - 14:18 kproc
root 97788 1 1 Jul 31 - 14:16 kproc
root 98044 1 0 Jul 31 - 14:30 kproc
root 98302 1 0 Jul 31 - 14:08 kproc
root 98306 1 0 Jul 31 - 14:10 kproc
root 98820 1 0 Jul 31 - 14:27 kproc
root 99078 1 0 Jul 31 - 14:17 kproc
root 99336 1 0 Jul 31 - 14:26 kproc
root 99594 1 0 Jul 31 - 14:26 kproc
root 99852 1 0 Jul 31 - 14:33 kproc
root 100110 1 0 Jul 31 - 14:31 kproc
root 100368 1 0 Jul 31 - 14:19 kproc
root 100626 1 0 Jul 31 - 14:24 kproc
root 100884 1 0 Jul 31 - 14:24 kproc
root 101142 1 0 Jul 31 - 14:13 kproc
root 101400 1 1 Jul 31 - 14:28 kproc
root 101658 1 0 Jul 31 - 14:15 kproc
root 101916 1 0 Jul 31 - 14:20 kproc
root 102174 1 0 Jul 31 - 14:41 kproc
root 102432 1 0 Jul 31 - 14:29 kproc
root 102690 1 0 Jul 31 - 14:22 kproc
root 102948 1 0 Jul 31 - 14:14 kproc
root 103208 1 0 Jul 31 - 14:18 kproc
root 103464 1 0 Jul 31 - 14:15 kproc
root 103722 1 0 Jul 31 - 14:16 kproc
root 103980 1 0 Jul 31 - 14:21 kproc
root 104238 1 0 Jul 31 - 14:19 kproc
root 104496 1 0 Jul 31 - 14:19 kproc
root 104754 1 0 Jul 31 - 14:30 kproc
root 105012 1 0 Jul 31 - 14:24 kproc
root 105270 1 0 Jul 31 - 14:11 kproc
root 105528 1 0 Jul 31 - 14:03 kproc
root 105786 1 0 Jul 31 - 14:15 kproc
root 106044 1 0 Jul 31 - 14:32 kproc
root 106304 1 1 Jul 31 - 14:13 kproc
root 106562 1 2 Jul 31 - 14:18 kproc
root 106818 1 0 Jul 31 - 14:16 kproc
root 107076 1 0 Jul 31 - 14:23 kproc
root 107334 1 0 Jul 31 - 14:22 kproc
root 107592 1 2 Jul 31 - 14:28 kproc
root 107850 1 0 Jul 31 - 14:25 kproc
root 108108 1 2 Jul 31 - 14:18 kproc
root 108366 1 0 Jul 31 - 14:10 kproc
root 108624 1 0 Jul 31 - 14:05 kproc
root 108882 1 0 Jul 31 - 14:25 kproc
root 109140 1 0 Jul 31 - 14:24 kproc
Thanks Again
Jorge
- Previous message: Fernandez Garay Jorge: "Performance problem"
- Maybe in reply to: Fernandez Garay Jorge: "Performance problem"
- Next in thread: Green, Simon: "Re: Performance problem"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
