I/O Performance Issues

From: Bryan Pepin (bpepin_at_comcast.net)
Date: 11/26/04

  • Next message: Alan Kong: "System can't bbot after removing SDS mirror" 
    To: <sunmanagers@sunmanagers.org>
Date: Thu, 25 Nov 2004 20:34:53 -0500

    Hello,

    Here is the issue. We recently performed a storage migration of a 1TB+
    Oracle DB running on a Sun E6500 Solaris 8 ( updated to KP 117350-12).
    It is a Data Warehouse application. As a part of the migration we also
    switched from JNI FC-64 SBUS HBA's to Emulex LP9002S HBA's ( quantity 4
    ). The new underlying storage infrastructure is all essentially
    perorming as expected and not presenting itself as any apparent
    bottleneck.

    The problem is, the system iowait times are huge since the migration,
    and we are trying to determine why the massive drop in I/O performance
    happened. We are now seeing I/O queue going to the devices and are
    suspicious of our emulex lpfc.conf setup.

    >From what I can tell, both JNI and Emulex recommend limiting
    sd_max_throttle and the lun_que lengths. One other difference is that we
    are now throttling our target queue's on the emulex cards, where as far
    as I can tell, JNI did not have that option.

    For the emulex cards, we essentially have 4 cards, each bound to 1
    target, and 122 data devices down each path. We are also running EMC
    PowerPath 4.x to enable I/O load balancing acrooss those cards.

    So what we're thinking of is disabling the target que restriction on the
    emulex adapters since we have so many luns down each target ( it is
    currently set at tgt-queue-depth=768). Also, at peak times we are seeing
    over 1000 I/O's per second down each Emulex card.

    One other piece of information is that during the migration, we pushed
    data at rates of over 300 GB/hour, so we know Emulex cards can perform
    well. We're thinking that our current settings though may be better
    designed for less I/O rates with larger size I/O, as opposed to very
    high I/O rates of smaller size.

    Does anyone have any experience/recommendations regarding this issue?

    Any help/advice would be appreciated.

    Will summarize.

    Thanks.

    -Bryan Pepin
    _______________________________________________
    sunmanagers mailing list
    sunmanagers@sunmanagers.org
    http://www.sunmanagers.org/mailman/listinfo/sunmanagers

  • Next message: Alan Kong: "System can't bbot after removing SDS mirror"