Re: What would be the best way to copy lots of files from oneserver to another.

From: Andy Dills (andy_at_xecu.net)
Date: 06/21/03

  • Next message: Simon: "Re: What would be the best way to copy lots of files from oneserver to another." 
    Date: Sat, 21 Jun 2003 11:26:28 -0400 (EDT)
To: Chuck Swiger <cswiger@mac.com>

    On Fri, 20 Jun 2003, Chuck Swiger wrote:

    > Etienne Ledoux wrote:
    > > I need to copy lots of directories/files from one server to another. Approx
    > > 45Gb . These files are mailboxes stored in maildir format. What would be the
    > > best/safest/quickest way to do this ?
    >
    > Putting the disks onto one machine and copying locally is going to be a win
    > considering the amount of data you have, but if you have to copy over the
    > network, consider using something like rsync.

    I love rsync and use it for lots of things.

    A one time move of 45 GB is not something I would use rsync for.

    Rsync is good for just what is says it's for: keeping thing in sync.

    Rsync will build metadata information about the ENTIRE data set before
    moving a single bit. This is very costly, making rsync's utility plummet
    as the data set size increases. You'll find it preferrable to write a
    script to traverse and descend directories, spawning numerous rsync
    processes for smaller sized directories.

    Of course, considering that it's a one-time move, I would advise
    physically moving the drive, to remove any sort of network involvement.

    I've been in your position before, and I can promise you that even using
    NFS and spawning multiple moves (mv /var/mail/a* /mnt &; mv /var/mail/b*
    /mnt &; etc.) will quickly outpace rsync.

    The number of concurrent processes involved in the move can really only be
    determined by watching the aggregate traffic to your server. You'll see it
    level off at a certain point (for me it was X=7), after which additional
    processes don't yield any more optimization.

    Andy 

    ---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
_______________________________________________
freebsd-isp@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-isp
To unsubscribe, send any mail to "freebsd-isp-unsubscribe@freebsd.org"
  • Next message: Simon: "Re: What would be the best way to copy lots of files from oneserver to another."