Re: Interesting speed benchmarks

Peter Jeremy <peterjeremy@xxxxxxxxxxxxxxxx> wrote:
On Sat, 2007-Jan-27 11:54:37 -0500, Brian Reichert wrote:
On Sat, Jan 27, 2007 at 09:43:52AM +1100, Peter Jeremy wrote:
Note that dump/restore is the only tool that can correctly reproduce
sparse files. tar, cpio and pax also have filename and file size
restrictions. I don't think that cpio or pax support ACLs or file
flags.

I thought 'star' handled sparse files and all the extra magic? Not
that this is germaine to the topic at hand...

You can detect a sparse file by comparing the length of the file
with the number of allocated blocks. You can't determine whether a
specific block is all NULLs or not allocated without either bypassing
the filesystem (which dump does) or using a syscall to retrieve a
bitmap of allocated blocks (which FreeBSD doesn't have). =20

Both rsync and gtar create a sparse file by not writing
blocks of zeroes given -S option. You are correct that there
is no special help for reading sparse files. I recall
pjd@xxxxxxxxxxx saying he plans to implement solaris specific
extensions to lseek() in zfs (lseek's whence arg can take
SEEK_HOLE/SEEK_DATA values, to seek to next null/nonnull
block). It would be nice if this was done for all
filesystems. Returning a bitmap is far more work than adding
SEEK_{HOLE,DATA}.

For backups I use vbackup & venti from plan9ports. Like dump
vbackup knows ufs1 & ufs2 structure to walk the filesytem and
write only live data to venti (a SHA1-addressed block storage
server) so it correctly handles sparse files. A full backup
is faster than dump. All subseqeuent backups are incremental
and generally run very fast (for example I can backup a UFS2
system at about 10MB/s to a USB2 ATA disk. dd shows write
speed of 23MB/s).

You can get at backed up data in a number of different ways:
1) vcat (makes an identical disk image given a snapshot's
`score')
2) vftp (you can browse/copy files individually)
3) vnfs + vmount (make one or more snapshots accessible in
filesystem namespace)

AFAIK flags, attributes & ACLs are maintained but are only
available if you use vcat. See http://swtch.com/plan9ports
_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: /usr/bin/cp preserving holes ?
    ... >> supports sparse files, but I don't see that it has anything to do with NFS ... >> some differences in the block count (due to differences in filesystem ... Absent a decent interface to determine hole location, ...
    (comp.unix.solaris)
  • Re: cp: I do not understand ...
    ... but did you try using backup/restore with the source filesystem ... You could try to find the sparse files on the source ... on the new filesystem instead of copying them over. ...
    (comp.unix.aix)
  • Re: O_DIRECT question
    ... there _is_ a problem, because they don't care. ... I don't regard it as a system failing that I am allowed to shoot myself ... Suppose you have a filesystem, not at all related to databases and stuff. ... "hope that sparse files are handled right" is a high hope. ...
    (Linux-Kernel)
  • Re: Incorrect file size?
    ... Filesystem Size Used Avail Capacity Mounted on ... /dev/ar0s1a on / (ufs, local) ... This is most likely due to the use of 'sparse files': ...
    (freebsd-stable)
  • Re: Interesting speed benchmarks
    ... I don't think that cpio or pax support ACLs or file ... I thought 'star' handled sparse files and all the extra magic? ... bitmap of allocated blocks. ...
    (freebsd-current)