Re: Faster backups - possibly Solaris pipe buffer size

Yonah Russ wrote:
Atro Tossavainen wrote:
"Yonah Russ" <yonah.russ@xxxxxxxxx> writes:
I want to speed up the backup- right now backing up 670GB+/- takes
close to 6.5 hours.
Averaging 30 MB/s doesn't sound half bad to me.


LTO3 is rated at 80MB/s- I don't know how realistic that is but I'd
like to get as close as possible- there is certainly room for
improvement.

Is there a reason you're using compression on the CPU instead of
letting the tape drive do its own?

The compression was in a backup script I inherited (one that was used
for a previous tape drive)

I also thought about getting rid of it but the daily backup of this
machine is critical so I can't take the chance that taking out the
compression will cause the backup to not fit on one tape.

As it is, dd only reports partial blocks on the side writing to the
tape so I have no clue how much it is actually writing.

The tape compression is also enabled so I imagine that trying to see
how much space is taken on the tape is not going to be helpful in
determining whether or not the lzo compression is worth anything.

Any ideas on how to speed up the process? believe the pipe buffer size
is hurting performance but I don't know of any way to change the size
of the buffer.
I have a hunch that your bottlenecks, if there are any, might be the
OpenSSL RC4 encryption and possibly the LZO compression on the Sun CPU.


Encryption is a must and RC4 seems to be one of the fastest algorithms
from what I see.

The LZO compression can change or even go away if I know that there is
something better or that it isn't necessary.

Thanks,

Yonah


--
Atro Tossavainen (Mr.) / The Institute of Biotechnology at
Systems Analyst, Techno-Amish & / the University of Helsinki, Finland,
+358-9-19158939 UNIX Dinosaur / employs me, but my opinions are my own.
< URL : http : / / www . helsinki . fi / %7E atossava / > NO FILE ATTACHMENTS

Regarding whether to do encryption on the cpu or the drive; My guess is
that once the data have been encrypted, there remain few "patterns" in
data for the drive to compress. To achieve any compression, it would
have to precede encryption. You can confirm as seen below.

So if the drive itself doesn't offer acceptable encryption, your best bet
may be to send Joerg Schilling an RFE for "star". :-).

% wc -c genunix 3313904
% bzip2 -c < genunix | wc -c 1145071
% openssl rc4 -e < genunix | wc -c 3313920
% openssl rc4 -e < genunix | bzip2 -c | wc -c 3328981
% bzip2 -c < genunix | openssl rc4 -e | wc -c 1145087
.

Relevant Pages

  • Re: IBM announces Encrypting tape drives
    ... CPU cycles from the main processor onto the device controller. ... "Why didn't IBM put the compression into the DASD/TAPE/whatever ... I would much rather have compression and encryption available on ... "Tape only" encryption solution greatly limits my choices. ...
    (bit.listserv.ibm-main)
  • Re: DAT/DDS tape backups...
    ... >DDS-4 drive complete with a 8 tape cassette autoloader. ... >At least that is how the specifications for these drives (and all ... >With compression turned off, I get 19459MB to tape, with it on, I get ... Don't feed the hardware compressor already ...
    (Fedora)
  • Re: DFHSM (and other tape) Encryption
    ... The obvious problem from the standpoint of efficiency is that good encryption of the data, which destroys apparent patterns in the data, will make tape hardware compression perform poorly. ... For a full volume dump of a 3390-3 the CPU time went from around 5 secs for uncompressed dump to around 38 secs for a compressed and encrypted dump, and that was with using the crypto engine. ...
    (bit.listserv.ibm-main)
  • Re: Why does measuring tape bend one way?
    ... > into tension and the edges into compression. ... > tape is thin compared to the length over which the compressive ... > slight curve with the energy released by the compressive force ... > moving as the deformation allows the force to move. ...
    (sci.physics)
  • Re: DFHSM (and other tape) Encryption
    ... The obvious problem from the standpoint of efficiency is that good encryption of the data, which destroys apparent patterns in the data, will make tape hardware compression perform poorly. ... We recently did a limited experiment with a software tool that can front-end DFDSS to encrypt dumps before they are written to a device. ...
    (bit.listserv.ibm-main)