Re: Mirroring 2 Machines with rsync

• Previous message: bazzz777: "Re: Mirroring 2 Machines with rsync"
• In reply to: Baby Peanut: "Re: Mirroring 2 Machines with rsync"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Date: 25 Jun 2003 16:36:34 -0700

baby_p_nut2@yahoo.com (Baby Peanut) wrote in message news:<96d83290.0306240738.60b84d6d@posting.google.com>...
> bazzz777@yahoo.com (bazzz777) wrote in message news:<6c9ad26f.0306230837.17005e63@posting.google.com>...
> > Anyone care to comment on the following?
> >
> > I have 2 rp2400 machines each with an Infotrend 500G array
> > and running HP-UX 11.11. I want one to be
> > a redundant mirrored copy of the other. Every night I sync them
> > up with the rsync utility, http://rsync.samba.org.
> >
> > So far, I exclude the following files:
> >
> > /etc/rc.config.d/netconf
> > /etc/utmp
> > /etc/opt/resmon
> > /dev
> >
> > Everything seems OK but rsync spits out a couple of
> > hundred unexplained errors every night.
> >
> > Anyone care to comment on this method of mirroring?
> > Are there more files I should be excluding?
> > Thanks for any observations, hints, suggestions, etc..
>
> I'm currently mirroring an L box to a K box. More than 5TB of data is
> moved but runs are only done twice a week since they take so long.
>
> The rsync utility has one big problem with scaling up: tables in core
> (RAM) memory that grow to huge sizes when there are huge numbers of
> files. I work around it by breaking up what rsync pushes into pieces
> that are pushed via separate invocations of rsync.
>
> I also push the rsync through OpenSSH to get compression. If you are
> syncing compressed stuff this may not be useful to you.
>
> Here is a shell script fragment that I modified to conceal the true
> nature of the real shell script that is run.
>
> dosync() {
> if [ ! -d $1 ]; then
> echo \\n$1 is not a directory, skipping.
> continue
> fi
>
> rsync -a --delete --stats --exclude=unwanted-patteren\* \
> --rsh="/usr/local/bin/ssh -C" \
> $i/ target-computer:$i \
> >> /var/log/rsync-log-file 2>&1
> }
>
> dosync /usr
> dosync /home
> etc.

I have 2 x 266 processors and 2G RAM in each machine. I do the
directories
separately as in "for x in $(ls /); do rsync <options> $x ....;done".
The largest filesystem/directory contains ~250,000 files. I wonder if
this is too many.

Baz

• Previous message: bazzz777: "Re: Mirroring 2 Machines with rsync"
• In reply to: Baby Peanut: "Re: Mirroring 2 Machines with rsync"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]