Bottleneck I/O or CPU

From: Chuck (hal___jordan_at_hotmail.com)
Date: 01/06/04 

Date: 6 Jan 2004 07:00:07 -0800

Given this output from vmstat and iostat, what would you say the
bottleneck is on this server? My main objective is to prove that it is
not I/O. Memory doesn't seem to be an issue either. Judging from
everything I've seen, the CPUs are the bottleneck.

I realize this data is affected by more variables than I can type into
this post, but I'm just looking for a general idea.

This output from these utilities was not performed at the same time,
but one after the other in about a five minute time span.

The server is an E4500 / 8 x 336MHz / 16gb memory
The storage is a few A5200s

VMSTAT

procs memory page disk faults
cpu
r b w swap free re mf pi po fr de sr m1 m2 m3 m5 in sy cs
us sy id
69 1 0 9899288 9361408 40 255 0 300 220 0 0 0 0 0 0 19531 151521
52700 34 65 1
88 1 0 9899880 9362856 62 276 0 12 8 0 0 0 0 0 0 16247 140865
53104 31 68 1
98 0 0 9896552 9359824 32 422 0 64 64 0 0 0 0 0 0 13326 144240
45238 36 64 0
98 1 0 9895560 9358240 38 182 0 160 148 0 0 0 0 0 0 9635 149757
39583 34 65 1
47 1 0 9894520 9357040 66 584 0 84 64 0 0 0 0 0 0 7699 143139
31124 36 63 0
86 1 0 9892280 9354208 4 187 0 92 92 0 0 0 0 0 0 8831 140926
37732 37 63 0
57 1 0 9890928 9352512 32 295 0 208 148 0 0 0 0 0 0 9065 140678
34023 36 63 0
58 5 0 9895784 9357264 73 488 28 192 168 0 0 20 0 2 4 7649 153151
35164 34 65 0
50 2 0 9891616 9356264 120 425 0 0 0 0 0 0 0 0 0 7210 168243
30245 36 63 1
98 1 0 9897072 9357688 165 840 0 100 100 0 0 0 0 1 0 8903 147784
34780 30 68 2
122 2 0 9898576 9358328 127 826 0 60 56 0 0 0 0 0 0 7587 145465
33405 32 66 2
111 2 0 9894112 9356944 62 246 0 24 24 0 0 0 0 0 0 10159 145313
40959 31 66 2
88 2 0 9897968 9356272 162 597 0 0 0 0 0 0 0 0 0 10091 142655
36416 30 68 2
130 2 0 9886352 9352992 7 123 0 0 0 0 0 0 0 0 0 8007 141771
35200 29 67 3
94 1 0 9885368 9351624 4 99 0 0 0 0 0 0 0 0 0 8119 140228
37063 30 68 3
54 2 0 9884760 9350552 1 4 0 100 100 0 0 0 0 0 0 5713 134647
35506 30 68 1
105 1 0 9886072 9351360 4 23 0 0 0 0 0 0 0 0 0 8109 140337
39404 29 69 3
110 2 0 9885624 9350944 1 95 0 0 0 0 0 0 0 0 0 6674 142586
33955 32 66 2
123 1 0 9891424 9352296 62 492 0 32 32 0 0 0 0 0 0 9185 144199
39785 31 67 2

IOSTAT ( I edited out all the disks with zero activity )

device r/s w/s kr/s kw/s wait actv svc_t %w %b
ssd1 0.0 0.8 0.0 6.4 0.0 0.0 6.5 0 1
ssd5 1.6 24.8 3.2 49.6 0.0 0.2 6.1 0 11
ssd8 0.0 21.2 0.0 42.4 0.0 0.1 3.5 0 7
ssd10 2.4 20.4 4.8 40.8 0.0 0.1 4.9 0 8
ssd11 91.0 0.0 182.0 0.0 0.0 0.0 0.4 0 4
ssd12 0.2 0.8 0.4 1.6 0.0 0.0 7.6 0 1
ssd13 0.0 1.8 0.0 16.0 0.0 0.0 6.7 0 1
ssd15 89.8 0.0 179.6 0.0 0.0 0.1 0.7 0 6
ssd16 89.4 0.0 178.8 0.0 0.0 0.0 0.6 0 5
ssd17 0.0 21.2 0.0 42.4 0.0 0.1 4.4 0 9
ssd19 13.6 24.8 27.2 49.6 0.0 0.1 2.9 0 9
ssd20 7.4 29.4 14.8 58.8 0.0 0.2 5.4 0 15
ssd24 0.0 11.8 0.0 23.6 0.0 0.1 4.8 0 5
ssd25 0.0 0.6 0.0 8.0 0.0 0.0 9.5 0 0
ssd27 0.0 0.2 0.0 1.6 0.0 0.0 9.9 0 0
ssd28 18.0 29.4 36.0 58.8 0.0 0.2 3.2 0 13
ssd30 2.8 7.6 5.6 15.2 0.0 0.1 5.3 0 5
ssd32 0.0 1.8 0.0 16.0 0.0 0.0 3.8 0 1
ssd40 16.2 20.6 32.4 41.2 0.0 0.1 2.5 0 8
ssd45 3.6 11.6 7.2 23.2 0.0 0.1 5.3 0 7
ssd46 104.8 0.0 209.6 0.0 0.0 0.1 0.5 0 5
ssd49 0.2 0.2 0.4 0.4 0.0 0.0 8.0 0 0
ssd52 0.0 1.0 0.0 16.0 0.0 0.0 9.8 0 1
ssd53 0.0 0.8 0.0 6.4 0.0 0.0 4.8 0 0
ssd54 141.0 0.0 282.0 0.0 0.0 0.1 0.4 0 5
ssd55 0.2 0.8 0.4 1.6 0.0 0.0 5.8 0 1
ssd62 98.8 0.0 197.6 0.0 0.0 0.1 0.6 0 6
ssd64 0.0 21.2 0.0 42.4 0.0 0.1 2.6 0 5
ssd65 101.0 0.0 202.0 0.0 0.0 0.1 0.5 0 5
ssd68 136.6 0.0 273.2 0.0 0.0 0.1 0.4 0 5
ssd76 3.6 19.8 7.2 39.6 0.0 0.1 6.2 0 11
ssd78 134.8 0.0 269.6 0.0 0.0 0.0 0.4 0 5
ssd81 0.0 0.4 0.0 3.2 0.0 0.0 6.4 0 0
ssd82 0.0 21.6 0.0 45.6 0.0 0.1 3.0 0 6
ssd87 0.0 11.6 0.0 23.2 0.0 0.0 3.1 0 3
ssd88 0.0 1.0 0.0 16.0 0.0 0.0 4.4 0 0
ssd89 0.0 0.2 0.0 7.6 0.0 0.0 5.4 0 0
ssd93 13.8 19.8 27.6 39.6 0.0 0.1 3.0 0 9
ssd94 0.0 0.2 0.0 1.6 0.0 0.0 8.1 0 0
ssd98 10.8 11.6 21.6 23.2 0.0 0.1 2.7 0 6
ssd101 135.2 0.0 270.4 0.0 0.0 0.0 0.4 0 5
ssd103 0.0 0.2 0.0 1.6 0.0 0.0 10.1 0 0
ssd106 0.0 0.6 0.0 8.0 0.0 0.0 7.5 0 0
ssd108 0.0 0.2 0.0 7.6 0.0 0.0 4.9 0 0
ssd109 0.0 0.2 0.0 1.6 0.0 0.0 3.9 0 0
ssd111 0.0 0.2 0.0 0.4 0.0 0.0 5.3 0 0
ssd112 134.8 0.0 269.6 0.0 0.0 0.0 0.4 0 5
ssd119 139.6 0.0 279.2 0.0 0.0 0.1 0.4 0 5
ssd129 10.2 7.6 20.4 15.2 0.0 0.0 2.6 0 4

Here is some sar output as well.

14:00:00 41 47 4 8
14:10:00 48 35 5 12
14:20:00 56 22 6 17
14:30:00 45 37 6 11
14:40:00 49 32 5 13
14:50:01 54 25 6 16
15:00:00 46 37 6 12
15:10:00 46 34 5 15
15:20:01 56 23 6 15
15:30:00 41 45 6 8
15:40:01 48 35 5 13
15:50:00 51 33 5 11
16:00:01 40 49 5 6

Thanks for any opinions on this.