Re: using star to move 3 millions files .. going slowly .. painfully
From: Joerg Schilling (js_at_cs.tu-berlin.de)
Date: 11/08/03
- Next message: Dr. David Kirkby: "Re: X server dies if one logs out."
- Previous message: Dennis Clarke: "using star to move 3 millions files .. going slowly .. painfully"
- In reply to: Dennis Clarke: "using star to move 3 millions files .. going slowly .. painfully"
- Next in thread: Dennis Clarke: "Re: using star to move 3 millions files .. going slowly .. painfully"
- Reply: Dennis Clarke: "Re: using star to move 3 millions files .. going slowly .. painfully"
- Reply: Logan Shaw: "Re: using star to move 3 millions files .. going slowly .. painfully"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: 8 Nov 2003 12:10:42 GMT
In article <Pine.GSO.4.58.0311080241480.28397@blastwave>,
Dennis Clarke <dclarke@blastwave.org> wrote:
>
>For reasons unknown I am getting pathetic performance out of star :
>
>bash-2.05$ star --version
>star 1.4.2 (sparc-sun-solaris2.8)
Well, star-1.4 is old and as the recent version is quite different from
previous releases, it makes no sense to run benchmarks using star-1.4.
....except when you are willing to edit the source and run a test with a
recompiled version as an important option is missing in star-1.4. You would
need to be able to check the equivalent of the -no-fsync option....
It would help to know which OS version you are using.
OK, below you tell us that this is S 9 with plenty of RAM.
Starting with the Solaris 8 update that did introduce the UFS filesystem
snapshots, several important speed ups that mainly affect your usage have
been introduced.
>I have two file systems on the same server. The source file system is moun=
>ted
>read only and is on two disks that are mirrored with two SCSI controllers.
>There should be no issue with performance there as the disks are 7200rpm SC=
>SI
>units ( Seagate 180Gb disks ) and the file system is UFS with normal Solari=
>s
>personality thus :
>
># df -t /var/oldmail
>/var/oldmail (/dev/md/dsk/d6 ): 179278372 blocks 80206435 files
> total: 333604696 blocks 83308160 files
As this looks like the AT&T variant of the df output, I assume 512 byte
"blocks" and we would need to manually subtract the top line from the
last line in addition. This would be 77163162 kB ==~ 76 GB and 3101725 files.
Thus the average file size is 24.877 kB. As this is a filesize where we
have no indirect blocks, nearly every file will use at least one fragment.
># mkfs -m /dev/md/dsk/d6
>mkfs -F ufs -o
>nsect=3D611,ntrack=3D24,bsize=3D8192,fragsize=3D2048,cgsize=3D6,free=3D1,rp=
>s=3D119,
> nbpi=3D2048,opt=3Dt,apc=3D0,gap=3D0,nrpos=3D8,maxcontig=3D128
> /dev/md/dsk/d6 354560856
On a fragmented filesystem, optimization for "time" is OK....
># mount -v | grep oldmail
>/dev/md/dsk/d6 on /var/oldmail type ufs
>read-only/setuid/intr/largefiles/logging/xattr/onerror=3Dpanic/dev=3D154000=
>6 on
>From my experience, the fastest write method in this case would be
not to use logging but to set the "fastfs" flag.
>There you will see that there are about 3 million files on /var/oldmail.
>
>Now I also have a destination for the pile of files. A filesystem that is =
>on
>dual A5200 Fibre arrays on dual controllers. There is a stripe set of elev=
>en
>FC-AL 10,000 rpm disks involved here, mirrored and duplexed. The file syst=
>em
>is mounted with ufslogging and there is nothing special about it other than=
> it
>should be a failrly quick place to dump files to. The server is a quad CPU
>E450 with 4Gb of RAM and the OS is on three way mirrored disks internally.
>
># df -t /var/mail
>/var/mail (/dev/md/dsk/d5 ): 172860772 blocks 45087755 files
> total: 182253488 blocks 45548992 files
Is this the the state of the target filesystem before you did start the copy?
This FS holds 4696358 kB ==~ 4.6 GB of data in 461237 files resulting in an
average filesize of 10.182 kB which I would call pretty fragmented...
Note that AFAIK Sun's UFS does not implement a data log but only logging for
the meta data. No fragemntation related issues are handled by turning on
logging.
>All I did to move the contents of /var/oldmail over to /var/mail was the
>following :
>
># cd /var/oldmail
># star -c -Hexustar -acl . | star -xp -acl -C /var/mail
The best method to do this would be either:
star -copy -p -acl -C /var/oldmail . /var/mail
or
star -copy -p -acl -no-fsync -C /var/oldmail . /var/mail
using star-1.5. If you have plenty of RAM, it may help to add e.g. fs=128M
If (as I would expect in your case) a striping process may cause extra penalties
in the write case. If you neither expect I/O errors during write nor filling up
the filesystem by another machine via NFS, it is apropriate to use -no-fsync.
>After almost three hours I see that there is less than 5Gb of data transfer=
>ed
>and less than 500,000 files out of the 3,000,000+ that I need to move.
>
>What could possibly be the issue here?
See above:
- Device I/O speed problems caused by many seeks.
This may become less if -no-fsync is used.
My test with extracting the FreeDB CD title database (av. file size is
1.5 kB) and a Linux kernel source tree result in a speed up by 20%
when -no-fsync is used.
- Time spend in computing the best position for the fragment blocks.
- Reading a large source directory because you use star-1.4
I don't know which version of Solaris you are using...
Newer versions together with a lot of RAM are better but star-1.5
reads all directories into allocated use space memory one chunk.
It is obviously not the speed of the (unneeded) pipe.
You did transfer 12438700 kB ==~ 12 GB if I assume the 500000 files being all
of the average size. This gives a speed of 11517.314 kB/s which is much below
the speed of a pipe on a recent machine.
But ~ 12 MB/s is not that bad for a lot of small files.
>It can't be device IO issues, there are six controllers on this server and
>none of them are in contention for this task. I benchmarked it at 53Mb/sec
>for a single file write and at 17.8Mb/sec for 17,000 file write test.
What was the average size for the 17,000 files?
>It can't be RAM or CPU as there are plenty of both.
>
>It can't be patches as I have the fibre drivers well patched on Sol9.
>
>Should I have used ufsdump? tar? cpio?
>From my tests. ufsdump is slower than star.
Sun's tar is much slower than star.
I expect cpio to give a performance between ufsdump and Sun's tar.
--
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
js@cs.tu-berlin.de (uni) If you don't have iso-8859-1
schilling@fokus.fraunhofer.de (work) chars I am J"org Schilling
URL: http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily
- Next message: Dr. David Kirkby: "Re: X server dies if one logs out."
- Previous message: Dennis Clarke: "using star to move 3 millions files .. going slowly .. painfully"
- In reply to: Dennis Clarke: "using star to move 3 millions files .. going slowly .. painfully"
- Next in thread: Dennis Clarke: "Re: using star to move 3 millions files .. going slowly .. painfully"
- Reply: Dennis Clarke: "Re: using star to move 3 millions files .. going slowly .. painfully"
- Reply: Logan Shaw: "Re: using star to move 3 millions files .. going slowly .. painfully"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
|
