Re: Disk performance issue

From: jeff barratt-mccartney (jbarratt_at_COMPSAT.COM)
Date: 08/19/04

  • Next message: Green, Simon: "Re: Disk performance issue"
    Date:         Wed, 18 Aug 2004 22:32:02 -0400
    To: aix-l@Princeton.EDU
    
    

    what are you backing up with? 8Mb/sec should never result in 90+ i/o wait.
    what does filemon say?

    -----Original Message-----
    From: IBM AIX Discussion List [mailto:aix-l@Princeton.EDU]On Behalf Of
    Bill Verzal
    Sent: Wednesday, August 18, 2004 1:45 PM
    To: aix-l@Princeton.EDU
    Subject: Disk performance issue

    Hi *,

    I am banging my head against a wall on this issue. Hopefully the
    collective knowledge can help me with this one.

    I have a EMC 3830 SYM. It is maxed out in terms of drives. All drives are
    RAID-1 mirrored. Half of the LUNs on the EMC box are Fibre connected (not
    via SAN, but point-to-point) and the other half are Timefinder BCV's, SCSI
    connected.

    When I backup the data to LTO tape (4 tapes needed), by the time I get to
    the last tape (or even if I run the 4th tape by itself), the performance of
    the backup to tape slows down by an order of magnitude. The previous 3
    tapes all run in about 2 hours each, the last tape runs in about 4-hours.

    Here is a snapshot of some performance commands:

    # symstat -c 3 -i 5

             DEVICE IO/sec KB/sec % Hits %Seq Num
    WP
    08:00:18 READ WRITE READ WRITE RD WRT READ
    Tracks

    08:00:24 00A2 (rhdisk43) 21 0 2666 0 14 N/A 33
    0
             00A3 (Not Visible ) 21 0 2677 0 10 N/A 33
    0
             00A4 (Not Visible ) 20 0 2656 0 5 N/A 25
    0
                                  ------ ------ ------ ------ --- --- ---
    ------
    Total 473 393 12771 4543 23 100 12
    1553

    08:00:30 00A2 (rhdisk43) 20 0 2634 0 5 N/A 10
    0
             00A3 (Not Visible ) 20 0 2709 0 5 N/A 30
    0
             00A4 (Not Visible ) 21 0 2666 0 10 N/A 24
    0
                                  ------ ------ ------ ------ --- --- ---
    ------
    Total 560 440 11407 4652 25 100 9
    938

    # iostat 3 3|grep -v 0.0

    tty: tin tout avg-cpu: % user % sys % idle %
    iowait

    Disks: % tm_act Kbps tps Kb_read Kb_wrtn
    hdisk43 94.2 7943.4 63.4 23936 0
    hdisk43 94.9 7633.1 60.3 22912 0
    hdisk43 96.6 7677.1 61.6 23044 0

    So, the EMC box is providing me with about 20 I/O's per second, yet the H80
    AIX box is reporting 94% I/O wait.

    EMC has been less than helpful on resolving the issue.

    Anyone see anything else ?

    AIX 5.2, ML-1
    H80 6-way, 8GB memory

    Thanks, Bill.

    --------------------------------------------------------

    There are only 10 types of people in the world: Those who understand
    binary, and those who don't ...

    Bill Verzal
    Project Leader, Komatsu America

    (847) 970-3726 - direct
    (847) 970-4184 - fax


  • Next message: Green, Simon: "Re: Disk performance issue"

    Relevant Pages

    • Disk performance issue
      ... I have a EMC 3830 SYM. ... When I backup the data to LTO tape, by the time I get to ... Here is a snapshot of some performance commands: ... So, the EMC box is providing me with about 20 I/O's per second, yet the H80 ...
      (AIX-L)
    • Re: Disk performance issue
      ... I have a EMC 3830 SYM. ... It is maxed out in terms of drives. ... When I backup the data to LTO tape, by the time I get to ...
      (AIX-L)
    • Re: Disk performance issue
      ... /var) to hold lists of filenames etc, is there any chance that this is ... I have a EMC 3830 SYM. ... When I backup the data to LTO tape, by the time I get to ...
      (AIX-L)
    • Re: Disk performance issue
      ... It is maxed out in terms of drives. ... Half of the LUNs on the EMC box are Fibre ... > the last tape, ... > the backup to tape slows down by an order of magnitude. ...
      (AIX-L)
    • Re: Virtual tape on VM
      ... you can create a nucleus extension that intercepts the tape ... I/O and transforms it in some way. ... In your new nucleus extenstion, you could direct that tape I/O to a CMS file ...
      (bit.listserv.ibm-main)