Shadow or Raid. - Don't do both! WAS Re: 306GB drives!

From: Mike Naime (mnaime_at_kc.rr.com)
Date: 08/23/03 

Date: Sat, 23 Aug 2003 01:51:56 GMT

Keith Parris <keithparris_NOSPAM@yahoo.com> wrote in message
news:cf15391e.0308210643.56f91f8f@posting.google.com...
> "Mike Naime" <mnaime@kc.rr.com> wrote in message
news:<%lj%a.93485$o27.2125405@twister.rdc-kc.rr.com>...
> > when you have a system crash. No 6+ hour performance hit
> > from all the shadow copy merges. One of the BIG complaints heard from
some
> > of our remote customers with LARGE databases.
>
> It is possible to minimize the pain of full-merge operations. Having
> done some research in this area recently, I thought it might be useful
> to post some of my findings:
> --------
> How to mitigate the pain of full merges (until host-based mini-merge
> feature is available):
>
> If possible, use MSCP controllers (HSJxx, HSD30/40/50), which have
> Volume Shadowing Assists, including the Write History Logging which
> allows mini-merges. But if you had those, you wouldn't be reading
> this, so...

Too small, and Too slow! Get real.

I had my first drive failure in our old CI cluster that uses HSJ's and
shadowing.
It froze the whole cluster instead of autosparing the disk!

On the HSG's a drive failure is a non-event for me.
I had a drive fail out thursday night. With ConsoleWorks, I got the
following:
1.) page/email that the drive failed.
2.) Page/Email that the Drive was removed from the storage set.
3.) Page/Email that the spareset drive was added.
I do not bother with notification for when the drives have normalized. The
first threee are enough for me.

I have not had a drive failure in the EVA yet.

>
> Avoid system crashes:

ROTFLMAO! That an easy one to say.
Why can't you just as easily "Avoid destroying your data center", or "Avoid
multiple Controller crashes"?
Short and simple hardware fails at some time or another that you cannot do
anything about. I'm all for using the latest technology to reduce my
workload. and system downtime.

> 1) Take crash dumps, find root causes, install patches.

We do, we did!, and we do when possible. Installing the "Latest" patches
can sometimes cause crashes! (RMS V300 ring any bells?)

A DS10 system that is used for a standby ORACLE database was trying to
shadow a 350GB disk. After we turned on shadowing, the system crashed.
Upon analysisi of the crash dump, it was determined that Shadowing was the
cause of the system crash!
Perhapse you can tell Michael and I what he did wrong in trying to shadow
the 350GB disk?
It worked on the ES45 that we tossed in to get it done. But the DS10 died
on every attempt!
We did not try a DS20.

> 2) Minimize the number of nodes in a cluster if possible, using fewer
> of more-powerful nodes instead of a lot of less-powerful nodes.

You need the more powerful nodes to handle the shadowing load! Not to run
the Application on it.

> Minimize the effect of crashes. Don't mount shadowsets on nodes which
> don't need to access them -- then if the node crashes, it won't start
> a merge on those shadowsets.
>
> Avoid disk I/Os. Use caching in the host as much as possible, to
> avoid having to do disk I/Os during a merge. Enable XFC. For RMS
> files, use Global Buffers. Use database caches.
>

XFC causes/caused Oracle corruption in 7.3. Have they fixed that yet in
7.3/7.3-1?
I can't get a strait answer on that question from either Oracle or HP.

> Get merges over and done with as fast as possible. (Don't prolong the
> pain.)

Why have the pain in the first place!!!
Trust the technology!

<Snow plow slice...>

> i) Don't combine large disks into controller-based stripesets or
> RAID-5 arrays and then shadow those huge units. Instead, shadow
> individual disks (or even smaller partitions of a disk). If you have
> to re-combine these into larger volumes for use up at the VMS level,
> do so with host-based RAID software, forming RAID 0+1 arrays.

Even more IO for the OS to handle!!!!

Basically you are stating here that shadowing and RAID technology DO NOT
MIX!!!!
Either SHADOW, or RAID. but DO NOT DO BOTH!
Why buy the raid array controller if I cannot use the technology???
Oh, I forgot... You need tons of spindles now to make all those
shadowsets.

> ii) Interestingly, empirical tests suggest that dividing up a single
> large disk into 4 partitions and shadowing those 4 partitions in
> parallel should allow merges of all 4 units in only 40% of the time
> (i.e. about 2.5 times as fast) as merging the disk as a single whole
> unit. Although the single disk head experiences seek contention among
> the 4 partitions, it's still much faster to complete the merge because
> of the 4 parallel merge threads.

I do not want to partitions drives. If it was the EVA I would make smaller
drives where necessary, but this takes up to much management time per
cluster. Especially when multiplied by 50+ clusters!

That is the miniscule amount of data compared to the Oracle Database.

> b) Don't use an entire disk if you don't need the space. Yes, the
> smallest disk you can buy new is 18 GB today, but you don't have to
> use the entire space. If you're using only 4 GB on a 36 GB disk,
> you'll suffer while the other 32 GB gets merged. Instead, create a 4
> GB partition and present that 4 GB volume to VMS. (Note that if you
> shadow multiple partitions on a Fibre Channel disk, you need to be
> running 7.3-1 or better.)

I do not want to partition! If I suddenly needed 5 gig tomorrow... you
would need to present another partition. Transfer the data. Shadow that
again! :-) Too much management.

>
> The Shadowing full-merge thread does 127-block I/Os from start to end
> of the disk, so read-ahead cache in controllers helps a lot, with
> close to 100% hit rates possible.
>
> Biggest adverse performance impact today seems to be on reads directed
> to areas of the disk not yet passed over by the merge fence, since
> those areas get re-merged for each and every I/O. (Impact of the
> merge thread itself seems to be lower, in my tests, perhaps because
> disks and controllers are so much faster now than when Shadowing first
> came out, and the 127-block I/Os which seemed huge in the early days
> get handled in a flash with today's fast Fibre Channel interconects
> and 15K RPM disks, especially with caching in the controllers.)
>

Funny, the client that was complaining about the LONG time (4-6 hours) for
shadow merges was running 9 or 18GB JBOD drives in 3 HSG80 cabinets on Fibre
channel interconnects that where presented to clustered GS160's You would
think that it would not be a problem since they seemed to be following all
of the criteria that you presented above.

Maybe this just does not scale to a larger setup!!!

> Try to place performance-critical files at the beginning of the LBN
> space on the disk, so they are first to be passed over by the merge
> fence. Consider creating smaller partitions at the beginning of disks
> for your performance-critical files. On today's Zoned-bit Recording
> disks, the first LBNs on the disk are the best-performing areas
> anyway, with the highest areal bit density (putting a given amount of
> data onto the fewest tracks to minimize seeks) and with the highest
> data transfer rates off the disk surface.
>
> More details are available in the presentation
> http://www2.openvms.org/kparris/s2003_volshad_perf.ppt
>
> > The EVA really is making it potentially possible to do away with
shadowing.
> > You do your backups and redundancy at the disk controller level. Not at
the
> > OS level.
>
> This strategy works fine until the controller [pair] fails, or the
> datacenter in which the EVA is located is destroyed by a disaster.

Lets be fair. This is true of your HBVS solution if you only have one data
center as well. And, if your data center is where your business is run,
Like in most companies. Your data center may be the least of your worries!

Example. Auto Assembly line: If the assembly line is trashed/destroyed.
Replacing the computers that run the line is miniscule compared to the cost
of the assembly line itself.

2 data centers connected by dark fibre. I will take EVA's over HBVS any
day! and do it on a smaller VMS box to boot!!!

Relevant Pages

  • Re: physical drive replacement
    ... > My first problem is determining which of the 2 physical drives in stripeset ... and mount it into the shadow set, which should trigger a shadow copy ... to swap out the bad disk and rebuild the stripeset, ...
    (comp.os.vms)
  • Re: physical drive replacement
    ... >> My first problem is determining which of the 2 physical drives in stripeset ... DELETE the disk, quiesce the bus and perform the physical swap, ... > and mount it into the shadow set, which should trigger a shadow copy ...
    (comp.os.vms)
  • Re: Implementing a RAID 1 Array
    ... For your RAID 1 drive is hardware RAID 1, you will not see the mirrored ... drive in the Disk Management console. ... The Volume Shadow Copy Service provides the backup infrastructure for the ...
    (microsoft.public.windows.server.sbs)
  • Re: Bad Shadow set member
    ... >>>>> If he just adds the disk with the system running and uses /CONFIRM, ... >>> drive to an already existing shadow set. ... mindful of JF comment about list the good member first ... The drives are identical, so maybe I do need to INIT afterall? ...
    (comp.os.vms)
  • Re: MinVMS or CD backup and volume shadowing
    ... shadow set with two data disks for the first time. ... Currently we have MinVMS on DISK1 (a data disk, ...
    (comp.os.vms)