Re: Determining cpu consumption

From: Mark Schlechte (mschlechte_at_REGINA.CA)
Date: 05/03/05

  • Next message: jstobby_at_CO.EL-DORADO.CA.US: "Judy Stobby is out of the office."
    Date:         Tue, 3 May 2005 10:53:50 -0600
    To: aix-l@Princeton.EDU
    
    

    Thanks as always Holger.

    >>> Holger.VanKoll@SWISSCOM.COM 05/03/05 10:41 am >>>
    tprof -x sleep 60
    will shed some light on kernel-usage
    others forms of trace, too
    also truss -P some_process_id
    might help

    I dont think you have very much faults, but the number of syscalls is very high. often caused by shellscripts with lots of pipes or other forms or process creation

    From: IBM AIX Discussion List [mailto:aix-l@Princeton.EDU] On Behalf Of Mark Schlechte
    Sent: Tuesday, May 03, 2005 6:31 PM
    To: aix-l@Princeton.EDU
    Subject: Determining cpu consumption

    Greetings,

    I'm looking for a little assistance on how to find what's consuming my cpu's.
    This is a dual processor 7029-6C3, 2GB ram.

    I posted about performance problems in that past and they have always been where I can see some I/O problems like a high hdisk utilization or my WA column being busy so I run vmstat, iostat or filemon.
    This time I have a different problem and my question is how do I determine what's running in sys and kernel mode when topas doesn't seem to reveal anything. Do I need nmon? Does my Paging Faults # seem kind of high?
    Some example outputs included.
                                                    Syscall 29903 Writech 25199
    Kernel 48.0 |############## | Reads 1240 Rawin 0
    User 52.0 |############### | Writes 124 Ttyout 397
    Wait 0.0 | | Forks 112 Igets 0
    Idle 0.0 | | Execs 139 Namei 13645
                                                    Runqueue 8.5 Dirblk 0
    Name PID CPU% PgSp Owner Waitqueue 0.0
    jre 581660 1.4 21.5 oracle
    perl 860244 0.7 3.6 oracle PAGING MEMORY
    topas 606356 0.7 2.0 root Faults 28764 Real,MB 2047
    perl 827412 0.7 1.4 oracle Steals 0 % Comp 60.2
    oidldapd 491774 0.5 27.9 oracle PgspIn 0 % Noncomp 39.8
    endpoint 180402 0.5 0.6 root PgspOut 0 % Client 6.2
    dtgreet 274630 0.3 1.7 root PageIn 0
    java 557072 0.2 47.8 oracle PageOut 3 PAGING SPACE
    oidldapd 487672 0.2 9.5 oracle Sios 2 Size,MB 4608
    syncd 77918 0.0 0.6 root % Used 32.0
    IBM.CSMAg 307352 0.0 2.1 root NFS (calls/sec) % Free 67.9
    emagent 1097738 0.0 55.9 oracle ServerV2 0
    oracle 598056 0.0 3.2 oracle ClientV2 0 Press:
    oracle 372918 0.0 3.2 oracle ServerV3 0 "h" for help
    java 552998 0.0 76.4 oracle ClientV3 0 "q" to quit
    gil 45078 0.0 0.1 root
    webcached 716910 0.0 9.0 oracle

    ROOT > vmstat 2
    System Configuration: lcpu=2 mem=2048MB
    kthr memory page faults cpu
    ----- ----------- ------------------------ ------------ -----------
     r b avm fre re pi po fr sr cy in sy cs us sy id wa
     3 1 566255 7297 0 0 0 27 89 0 1389 23549 4213 9 19 70 2
     1 0 566248 7303 0 0 0 0 0 0 1245 57666 3423 32 34 34 0
     3 0 566746 6803 0 0 0 0 0 0 1208 46799 3523 23 48 29 0
     6 0 567849 5700 0 0 0 0 0 0 998 72229 2920 16 67 17 0
    11 0 567989 5547 0 0 0 0 0 0 723 49194 2089 18 82 0 0
     8 0 567520 6016 0 0 0 0 0 0 885 41116 2298 36 64 0 0
     6 0 567488 6047 0 0 0 0 0 0 1072 30745 2678 56 43 1 0
     5 0 567716 5820 0 0 0 0 0 0 907 39524 2102 42 58 0 0
     6 0 566087 7449 0 0 0 0 0 0 1199 46564 2940 28 64 8 0
     3 0 567220 6316 0 0 0 0 0 0 1298 43683 3699 16 57 27 0
     3 0 567189 6347 0 0 0 0 0 0 1234 37332 3671 20 58 22 0
     1 0 567308 6227 0 0 0 0 0 0 1298 24926 3640 28 38 33 0
     5 0 567195 6340 0 0 0 0 0 0 1464 22643 3815 31 33 35 1
     1 0 565715 7820 0 0 0 0 0 0 1587 31954 4168 21 46 33 0
     0 0 565715 7820 0 0 0 0 0 0 1471 6298 4743 1 3 96 0
     5 0 566400 7134 0 0 0 0 0 0 1521 77401 4479 11 17 71 2
     2 0 565715 7819 0 0 0 0 0 0 1740 13414 4856 9 12 78 1
     0 0 566660 6909 0 0 0 0 0 0 1533 44926 4725 7 5 88 0
     6 0 565715 7816 0 0 0 0 0 0 1470 8164 4650 4 3 93 0
     1 0 565715 7814 0 0 0 0 0 0 1657 11106 4866 5 2 92 1
    17:ROOT > topten
    Tue May 3 10:20:28 CST 2005
      10:20AM up 23 days, 51 mins, 6 users, load average: 4.50, 3.71, 3.40

         TOP TEN SORTED BY CPU USE
    USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
    root 12294 38.1 0.0 48 40 - A Apr 10 25300:42 wait
    root 8196 33.9 0.0 48 44 - A Apr 10 22486:52 wait
    oracle 581660 1.3 0.0 22032 8344 - A Apr 10 885:09 /app/oracle/jre
    oracle 491774 0.9 0.0 36276 600 - A Apr 10 613:39 oidldapd
    oracle 1097738 0.3 1.0 57320 28780 - A Apr 28 42:53 /app/oracle/pro
    root 77918 0.2 0.0 640 264 - A Apr 10 142:16 /usr/sbin/syncd
    sde 1126512 0.2 1.0 10464 10496 - A 08:39:22 0:23 /sde/v9.0/cgisp
    oracle 598056 0.1 1.0 44316 14840 - A Apr 10 34:34 oracleiasdb (LO
    root 180402 0.1 0.0 1304 296 - A Apr 10 63:29 /usr/lpp/NetIQ/
    root 274630 0.1 0.0 1764 256 - A Apr 10 94:59 dtgreet

    I think it may be related to some runaway oracle process (like PID 581660) but I can't figure out how to track that down.

    Mark

    DISCLAIMER: The information transmitted is intended only
    for the addressee and may contain confidential,
    proprietary and/or privileged material. Any
    unauthorized review, distribution or other use
    of or the taking of any action in reliance upon
    this information is prohibited. If you received
    this in error, please contact the sender and
    delete or destroy this message and any copies.

    DISCLAIMER: The information transmitted is intended only
    for the addressee and may contain confidential,
    proprietary and/or privileged material. Any
    unauthorized review, distribution or other use
    of or the taking of any action in reliance upon
    this information is prohibited. If you received
    this in error, please contact the sender and
    delete or destroy this message and any copies.


  • Next message: jstobby_at_CO.EL-DORADO.CA.US: "Judy Stobby is out of the office."

    Relevant Pages

    • Re: date -1 question
      ... unauthorized review, distribution or other use ... of or the taking of any action in reliance upon ... please contact the sender and ...
      (AIX-L)
    • Re: ksh script question
      ... for the addressee and may contain confidential, ... unauthorized review, distribution or other use ... of or the taking of any action in reliance upon ...
      (AIX-L)