Re: Bottleneck I/O or CPU

From: Joseph A. Rich (jar_at_nospam.teamquest.com)
Date: 01/06/04 

Date: Tue, 06 Jan 2004 19:13:41 GMT

Looks like a CPU bottleneck to me. You're funning flat out with a large run
queue. VMSTAT shows most of the CPU time is being spent in OS for the
processes, but the sar output is showing slightly the opposite. I/O is
negligble.

Joe

In article <b1713487.0401060700.67801e1a@posting.google.com>,
hal___jordan@hotmail.com (Chuck) wrote:
>Given this output from vmstat and iostat, what would you say the
>bottleneck is on this server? My main objective is to prove that it is
>not I/O. Memory doesn't seem to be an issue either. Judging from
>everything I've seen, the CPUs are the bottleneck.
>
>I realize this data is affected by more variables than I can type into
>this post, but I'm just looking for a general idea.
>
>This output from these utilities was not performed at the same time,
>but one after the other in about a five minute time span.
>
>The server is an E4500 / 8 x 336MHz / 16gb memory
>The storage is a few A5200s
>
>VMSTAT
>
>procs memory page disk faults
>cpu
>r b w swap free re mf pi po fr de sr m1 m2 m3 m5 in sy cs
>us sy id
>69 1 0 9899288 9361408 40 255 0 300 220 0 0 0 0 0 0 19531 151521
>52700 34 65 1
>88 1 0 9899880 9362856 62 276 0 12 8 0 0 0 0 0 0 16247 140865
>53104 31 68 1
>98 0 0 9896552 9359824 32 422 0 64 64 0 0 0 0 0 0 13326 144240
>45238 36 64 0
>98 1 0 9895560 9358240 38 182 0 160 148 0 0 0 0 0 0 9635 149757
>39583 34 65 1
>47 1 0 9894520 9357040 66 584 0 84 64 0 0 0 0 0 0 7699 143139
>31124 36 63 0
>86 1 0 9892280 9354208 4 187 0 92 92 0 0 0 0 0 0 8831 140926
>37732 37 63 0
>57 1 0 9890928 9352512 32 295 0 208 148 0 0 0 0 0 0 9065 140678
>34023 36 63 0
>58 5 0 9895784 9357264 73 488 28 192 168 0 0 20 0 2 4 7649 153151
>35164 34 65 0
>50 2 0 9891616 9356264 120 425 0 0 0 0 0 0 0 0 0 7210 168243
>30245 36 63 1
>98 1 0 9897072 9357688 165 840 0 100 100 0 0 0 0 1 0 8903 147784
>34780 30 68 2
>122 2 0 9898576 9358328 127 826 0 60 56 0 0 0 0 0 0 7587 145465
>33405 32 66 2
>111 2 0 9894112 9356944 62 246 0 24 24 0 0 0 0 0 0 10159 145313
>40959 31 66 2
>88 2 0 9897968 9356272 162 597 0 0 0 0 0 0 0 0 0 10091 142655
>36416 30 68 2
>130 2 0 9886352 9352992 7 123 0 0 0 0 0 0 0 0 0 8007 141771
>35200 29 67 3
>94 1 0 9885368 9351624 4 99 0 0 0 0 0 0 0 0 0 8119 140228
>37063 30 68 3
>54 2 0 9884760 9350552 1 4 0 100 100 0 0 0 0 0 0 5713 134647
>35506 30 68 1
>105 1 0 9886072 9351360 4 23 0 0 0 0 0 0 0 0 0 8109 140337
>39404 29 69 3
>110 2 0 9885624 9350944 1 95 0 0 0 0 0 0 0 0 0 6674 142586
>33955 32 66 2
>123 1 0 9891424 9352296 62 492 0 32 32 0 0 0 0 0 0 9185 144199
>39785 31 67 2
>
>IOSTAT ( I edited out all the disks with zero activity )
>
>device r/s w/s kr/s kw/s wait actv svc_t %w %b
>ssd1 0.0 0.8 0.0 6.4 0.0 0.0 6.5 0 1
>ssd5 1.6 24.8 3.2 49.6 0.0 0.2 6.1 0 11
>ssd8 0.0 21.2 0.0 42.4 0.0 0.1 3.5 0 7
>ssd10 2.4 20.4 4.8 40.8 0.0 0.1 4.9 0 8
>ssd11 91.0 0.0 182.0 0.0 0.0 0.0 0.4 0 4
>ssd12 0.2 0.8 0.4 1.6 0.0 0.0 7.6 0 1
>ssd13 0.0 1.8 0.0 16.0 0.0 0.0 6.7 0 1
>ssd15 89.8 0.0 179.6 0.0 0.0 0.1 0.7 0 6
>ssd16 89.4 0.0 178.8 0.0 0.0 0.0 0.6 0 5
>ssd17 0.0 21.2 0.0 42.4 0.0 0.1 4.4 0 9
>ssd19 13.6 24.8 27.2 49.6 0.0 0.1 2.9 0 9
>ssd20 7.4 29.4 14.8 58.8 0.0 0.2 5.4 0 15
>ssd24 0.0 11.8 0.0 23.6 0.0 0.1 4.8 0 5
>ssd25 0.0 0.6 0.0 8.0 0.0 0.0 9.5 0 0
>ssd27 0.0 0.2 0.0 1.6 0.0 0.0 9.9 0 0
>ssd28 18.0 29.4 36.0 58.8 0.0 0.2 3.2 0 13
>ssd30 2.8 7.6 5.6 15.2 0.0 0.1 5.3 0 5
>ssd32 0.0 1.8 0.0 16.0 0.0 0.0 3.8 0 1
>ssd40 16.2 20.6 32.4 41.2 0.0 0.1 2.5 0 8
>ssd45 3.6 11.6 7.2 23.2 0.0 0.1 5.3 0 7
>ssd46 104.8 0.0 209.6 0.0 0.0 0.1 0.5 0 5
>ssd49 0.2 0.2 0.4 0.4 0.0 0.0 8.0 0 0
>ssd52 0.0 1.0 0.0 16.0 0.0 0.0 9.8 0 1
>ssd53 0.0 0.8 0.0 6.4 0.0 0.0 4.8 0 0
>ssd54 141.0 0.0 282.0 0.0 0.0 0.1 0.4 0 5
>ssd55 0.2 0.8 0.4 1.6 0.0 0.0 5.8 0 1
>ssd62 98.8 0.0 197.6 0.0 0.0 0.1 0.6 0 6
>ssd64 0.0 21.2 0.0 42.4 0.0 0.1 2.6 0 5
>ssd65 101.0 0.0 202.0 0.0 0.0 0.1 0.5 0 5
>ssd68 136.6 0.0 273.2 0.0 0.0 0.1 0.4 0 5
>ssd76 3.6 19.8 7.2 39.6 0.0 0.1 6.2 0 11
>ssd78 134.8 0.0 269.6 0.0 0.0 0.0 0.4 0 5
>ssd81 0.0 0.4 0.0 3.2 0.0 0.0 6.4 0 0
>ssd82 0.0 21.6 0.0 45.6 0.0 0.1 3.0 0 6
>ssd87 0.0 11.6 0.0 23.2 0.0 0.0 3.1 0 3
>ssd88 0.0 1.0 0.0 16.0 0.0 0.0 4.4 0 0
>ssd89 0.0 0.2 0.0 7.6 0.0 0.0 5.4 0 0
>ssd93 13.8 19.8 27.6 39.6 0.0 0.1 3.0 0 9
>ssd94 0.0 0.2 0.0 1.6 0.0 0.0 8.1 0 0
>ssd98 10.8 11.6 21.6 23.2 0.0 0.1 2.7 0 6
>ssd101 135.2 0.0 270.4 0.0 0.0 0.0 0.4 0 5
>ssd103 0.0 0.2 0.0 1.6 0.0 0.0 10.1 0 0
>ssd106 0.0 0.6 0.0 8.0 0.0 0.0 7.5 0 0
>ssd108 0.0 0.2 0.0 7.6 0.0 0.0 4.9 0 0
>ssd109 0.0 0.2 0.0 1.6 0.0 0.0 3.9 0 0
>ssd111 0.0 0.2 0.0 0.4 0.0 0.0 5.3 0 0
>ssd112 134.8 0.0 269.6 0.0 0.0 0.0 0.4 0 5
>ssd119 139.6 0.0 279.2 0.0 0.0 0.1 0.4 0 5
>ssd129 10.2 7.6 20.4 15.2 0.0 0.0 2.6 0 4
>
>Here is some sar output as well.
>
>14:00:00 41 47 4 8
>14:10:00 48 35 5 12
>14:20:00 56 22 6 17
>14:30:00 45 37 6 11
>14:40:00 49 32 5 13
>14:50:01 54 25 6 16
>15:00:00 46 37 6 12
>15:10:00 46 34 5 15
>15:20:01 56 23 6 15
>15:30:00 41 45 6 8
>15:40:01 48 35 5 13
>15:50:00 51 33 5 11
>16:00:01 40 49 5 6
>
>Thanks for any opinions on this.