Re: Performance Tracker project update

On Wed, Jan 30, 2008 at 07:20:23PM +0100, Erik Cederstrand wrote:
Kris Kennaway wrote:
Robert Watson wrote:
One thing I am looking at is how to best create a library of world
tarballs that can be used to populate a nfsroot (or hybrid of periodic
tarballs + binary diffs to save space). Then you could provide your
benchmark in a standardized format (start/end/cleanup scripts, etc) and
tell a machine "go and run this benchmark on every daily snapshot for the
last year and give me the numbers".

Actually, this just bit me. To make packages for installation, I do a
standard installworld/kernel/distribution into a directory and a chroot
install of the necessary ports. I create a .tgz and the end result is
around 90MB (takes ca. 2 hours). Now I have 186 of those, and my file
system is full (it's only 15GB, but even a 500GB disk will fill up in 2-3
months with a fast build server).

I'd like a situation where I can very quickly set up a slave with a
specific version of FreeBSD to run additional tests or provide shell access
to a developer. This currently involves adding an entry to a queue,
rebooting and waiting 2 minutes. Quick and easy, but the archiving strategy
is obviously very inefficient.

I'm thinking of a couple of options:
1. Having one full install per month and archiving the rest as diffs
against that by recursively bsdiff'ing every file in the tree (I
could bsdiff a whole tarball, but bsdiff is very memory-intensive).
Quick test: 25 mins.
2. Make a hash of all files and only store the binaries where the hash
is different from the monthly tarball. Faster than 1., but less
effective. Quick test: 5 mins.
3. Use some kind of VCS. My experience with Subversion and binary files
is that it's very slow.
4. Throw hardware at the problem.

I'd say it should not take more than 10 mins to recreate an archived
version. Any thoughts?

It seems like you should be able to combine 1 and 2 with checksums to
decide if you need to run diffs. I'd think that would be quite fast.

Another option would be to use ZFS snapshots. As long as you avoid replacing
identical files unnecessicairly, you should have pretty good effencies and
little overhead.

-- Brooks

Attachment: pgp2bYik0TiNS.pgp
Description: PGP signature

Relevant Pages

  • Re: Performance Tracker project update
    ... One thing I am looking at is how to best create a library of world tarballs that can be used to populate a nfsroot. ... To make packages for installation, I do a standard installworld/kernel/distribution into a directory and a chroot install of the necessary ports. ... but the archiving strategy is obviously very inefficient. ... Quick test: 25 mins. ...
    (freebsd-performance)
  • Re: [opensuse] tarballs
    ... Joe and Peter and I think there's another fellow named ... Like checking out security issues on Suse next! ... |> know if there is a way to install tarballs properly in Suse. ... On crappy tarballs you might need to do 'sudo make install' before ...
    (SuSE)
  • Re: [opensuse] tarballs
    ... Joe Sloan schrieb: ... | John B Pace wrote: ... |> know if there is a way to install tarballs properly in Suse. ... On crappy tarballs you might need to do 'sudo make install' before ...
    (SuSE)
  • Re: choosing a distribution
    ... Slack packages are not tarballs in the sense I used it, ... and some "Install Shield" wizard started up. ... many Linux users could not care ...
    (alt.os.linux)
  • Re: eggs considered harmful
    ... tarballs and put them in a depot and b) other users get tarballs from ... python packages played well in this context. ... Eggs and with other setuptools-inspired install processes break this ...
    (comp.lang.python)