Re: BACKUP locks up a directory?


Dave Froble wrote:
AEF wrote:
Dave Froble wrote:
AEF wrote:
Richard B. Gilbert wrote:
AEF wrote:
[...]
If you really need to have all 24,000 files on line, I'd suggest
spreading them over a LOT more directories. Maybe a directory for 2006,
one for 2005, 2004, . . . or a directory for each month or whatever.
Thanks for your quick response.

Performance is not really an issue.
You indicate that the BACKUP performance is an issue. Your statement is
wrong.

Well, it is and it isn't. I have A LOT OF TROUBLE with these blasted
TLZ tape drives. (I'm the one who wrote "I Lost It with the Tape
Drive".) Currently, I'm happy if I just end up with tapes that I can
read back later. Sure, I'd like it to go faster, but that's the least
of my problems. In fact, I switched from DDS-2 tapes to DDS-1 tapes on
the hope that it would it would greatly increase the likelihood of
making tapes that are readable when I need them later. Until this
episode, it seemed to be working, which is ANOTHER problem I'm having.
The stupid drive is having trouble verifying the tape it just made!
Arrrggh. The problem here is that the backup somehow screwed up the
file-copy operation requiring me to rerun the EOD job (which because of
other screwy stuff forces me to manually run and adjust the report job,
but never mind that now) and I'd just like to understand better what's
going on.

4 MM DAT was a crap shoot since the day it first came out. That's just
the way things are. There have been many posts in the past, with the
usual recommendation to pick up a DLT drive. DLT4 and subsequent models
store significant data, and suffer many more passes than 4 MM DAT. I
understand new tapes for the older drives can be an issue.

I wish I could get DLT drives, but my VAX systems and the trading
system that runs on them is to be phased out. :-( I'm not going to get
the funds to get DLT drives. One thing, perhaps, that could save the
day is to NetBackup to our existing DLT drives. If things continue
going sour, I think I'll explore that option.

Are you saying getting DDS-1 tapes is problematic? Well, our supplier
had to back-order my favorite brand recently. And if the DDS-1 format
were discontinued, our Stratus would also be in trouble. Its drive uses
ONLY DDS-1. My company wants to phase that out, too, but we have many
desks using it currently.


The files normally copy over rather
quickly even though the directory is so huge. I suspect that BACKUP
started reading the .DIR file in the middle of the copy operation.
Could it be that the copying instigated a .DIR-file expansion and the
shuffling of the directory data screwed up the copy? Or does it have
something to do with BACKUP? (If I needed to delete selected files,
that would certainly go rather slowly!!!)

Actually, I suppose the backup would run faster if I combined each
day's files into BACKUP save sets. I actually do that before I archive
them to CD's, but I prefer to have the individual files available
online, at least for stuff this year (I need the current year's files
in order to run occasional usage reports).

Re spreading files over more directories -- I guess that would greatly
reduce the odds of this problem happenning again. But it would be more
work than it's worth.
No, it's not. If you have any method of identifying files by time, then
the work is rather trivial. Retrival may also require some work, but

Retrival? What is that?

defining a logical with a list of values handles that. A directory that
large is obscene even on VMS V7.2 and later. On V6.2 it's very much
worse. You've been given excellent advice and you've decided to ignore
it. Why did you ask?

Still pissed off at me, huh, aren't you Dave? Anyway, I didn't ignore
the answer, I just commented on it.

?????????

When was I ever upset with you. I know I'm getting old, crotchety, and
senile, but the only person I remember that caused frothing at the mouth
was JF, and now I don't let him bother me.

Maybe I'm thinking of someone else. Maybe I'm thinking about another
Dave. Maybe it was the way you wrote about my "ignoring a solution".
Sorry.


Large directories on V6.2 are a greater problem than just what you see
on the surface. It's not just deletions that cause problems, though
those are rather visable.

Well, if this problem becomes a recurring one or if I get to, somehow,
understand better what's going on, then I'll consider the
multiple-directory solution. But with a 2751-block directory, getting
it down to 128 blocks or less per new directory would require 22
directories! To keep things simple, I'd have to have dedicated
directory for each month's worth of data. With more than three years'
worth of data for two markets, that's going to be 72 directories just
for 2003 - 2005! That will take a while to set up!!!

One think I'd prefer to do, perhaps, is to put each month's data into a
save set and extract any currently needed files to a scratch area. I'm
doing that now on an as needed basis (as needed to keep the disk from
filling up!) when I archive them. The files consist of data files and a
save set contiaining log files, so for each month (for each desk) I
copy an archive save set and the log-files save set to the CD's.


Anyway**2, it's not worth the trouble because I've been doing this for
the last 5 years or so and have only once had this problem. It is far
easier to rerun and end-of-day once every 5 years than to rewrite the
EOD code to deal with a new-fangled directory structure and QA it and
get management approval and install it and debug it. I'd also have to
rewrite some other code that moves each day's files into a single save
set for transference to a CD-R. I'd also have to rewrite my
usage-report command procedures. OK, that wouldn't be too hard given
logical names.

Yes, the directories are rather large. But it's not really a problem
because I am the only one who ever uses these files. If I had to delete
them, there are several tricks available to speed it up. WIldcard
lookups are a bit slow, but I can do other things while they run. Why
do I even have these file online? I was told that they should be made
available just in case someone needed them for auditing purposes or
checking something or whatever. Guess how many times that's happened?
ZERO! And that covers at least 5 years. I'm also supposed to archive
them all to CD's, but that's a real pain, and I've read that CD-R's may
not last more than a few years. Taping is far easier except that the
TLZ tape drives make write-probably/read-maybe tapes!

Your comments may be applicalbe in other situations, but I really
didn't want to go into so much deatil.

Why did I ask? I didn't ask the question whose answer you speak about.
I asked if BACKUP locking up a directory is normal behavior or if
something else is happening. 7

I'd be real surprised if BACKUP locked a directory, unless you used the
/RECORD switch (I think that's the one that sets the last backup date).

Yes, I used /RECORD. Why would that lock the directory? I think that
isn't the case because of the order in which the recording pass is
done. It goes in (perhaps approximately) reverse backup order, which is
not a directory at a time but a continual traversing up and down the
levels of subdirectory-space (or directory-tree) space. (Look at a
BACKUP save set listing to see the order.) But even if it is, this
problem happened during the save phase.

It's mainly a read only operation. However, if it was reading, and
another process wanted exclusive access, that might be an issue. Don't
know enough about the details to be more specific.

Well, maybe next week someone can tell us more.


Not picking on anyone, just emphasizing that very large directory files
are a serious problem.

OK. Thanks for your help. All friendly advice is welcome.


--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@xxxxxxxxxxxxx
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

AEF

.

Relevant Pages

  • Re: Offsite backup options
    ... I like Exabyte VXA Drives for this. ... are sized and priced for what you need, and if you outgrow the tapes you can ... Any backup to a NAS, storage server, home server, or whatever seems like ... Can only do critical data ...
    (microsoft.public.windows.server.sbs)
  • Re: Anyone using Iomega REV for SBS Backup?
    ... are there any issues/gotchas with using SBS Backup for restore ... We use Rev drives on lots of our SBS installations after experiencing lots ... install any of the Iomega drivers and just use Filestreamer RM. ... not so costly for DAT but Travan tapes are very expensive. ...
    (microsoft.public.windows.server.sbs)
  • Re: Need backup hardware suggestions
    ... > I am exploring various solutions for reliable backup for my exclusively ... > Tapes or external USB harddisks? ... I happen to prefer Exabyte VXA-2 tape drives, ... Since my initial installation of the VXA-1 drive, ...
    (comp.os.linux.misc)
  • Re: Need backup hardware suggestions
    ... > I am exploring various solutions for reliable backup for my exclusively ... > Tapes or external USB harddisks? ... I happen to prefer Exabyte VXA-2 tape drives, ... Since my initial installation of the VXA-1 drive, ...
    (comp.os.linux.hardware)
  • Re: DLT 4000 (DLT III and DLT IV Tapes)
    ... Yes -- a bonfide Backup application such as Arcserve. ... Dave ... 2000 Server can format these DLT III tapes? ...
    (microsoft.public.win2000.general)