Re: Which delete statement is faster?

In article <13ha3tiakjblj1b@xxxxxxxxxxxxxxxxxx>, "Mike Minor" <mminorhsd@xxxxxxxxxxxxx> writes:
I have a directory with 200000+files, all in the a*.txt;1 range. I need to
ftp these files to another server. After sending 30,000+ files via FTP I
realized the magnatude of the ftp process, and interrupted it. I want to
delete the 30,000+ file already ftp'ed before going back and looking at
continueing the ftp process in a different manner and it just seems to be
taking an extremely long time to perform the delete. I think the hang up is
the re-write of the directory contents back to disk after a few files are
deleted. I did the delete with a /log to watch how long it took to delete a
file. I noticed a pause of a few seconds after it listed 15 to 20 files that
were deleted.....

Top posting. *sigh*.

In any case, you've hit the "bubble down" problem that I alluded to.
And, because you want to delete the first 30,000 files in a 200,000
file directory, reverse alphabetical order isn't going to do
much for you.

Hmmmm...

There _is_ a sneaky approach.

How about if instead of

$ delete a*.txt;1

you

$ rename a*.txt;1 *.*;2 /log

That should run pretty fast because the directory entries can be
modified in place. There won't be any "bubble down".

You can press control-Y when you come to the 30,000th file.

Then you can go back in and ftp all of your remaining version 1 files.

ftp> mput a*.txt;1

When you're finished you can do a reverse alphabetical order delete
on the whole directory.

<briggs@xxxxxxxxxxxxxxxxx> wrote in message
news:iO4j523y6UmT@xxxxxxxxxxxxxxxxxxxxxxxxxxx
In article <13h9vel51sekc1b@xxxxxxxxxxxxxxxxxx>, "Mike Minor"
<mminorhsd@xxxxxxxxxxxxx> writes:
Is there any difference in the speed at which the command is executed in
the
following examples?

del a*.*;*
del a*.txt;*
del a*.txt;1

All things being equal (i.e. the only files in the directory that
match the a*.*;* wildcard also match a*.txt;1), I'd expect no significant
performance difference.

The real work is going to be the disk I/O writing directory contents
back to disk. Reading directory entries into cache and parsing
and searching directory entries from cache is unlikely to be the
bottleneck.

Why do you ask?

Historically, the thing that absolutely kills delete performance is
the "bubble down" that can take place if you delete the last directory
entry in a block near the front end of a _HUGE_ .DIR file.

Various tweaks over the years have improved this behavior by orders
of magnitude. If it's still an issue for you, a
reverse-alphabetical-order
delete is one thing that can sometimes be of use.


.

Relevant Pages

  • Re: JINETD Job stuck will not abort?
    ... What version of MPE, FTP, Network, etc? ... Is the network clean? ... JINETD Job stuck will not abort? ... This inbound FTP process seems to be doing a loop for each ...
    (comp.sys.hp.mpe)
  • Help: Large Incoming FTP Files Using Entire Server Memory
    ... I'm scripting an FTP process to backup large disk image files from a Linux ... degrades performance on the Windows server. ...
    (microsoft.public.inetserver.iis.ftp)
  • Re: Which delete statement is faster?
    ... After sending 30,000+ files via FTP I ... realized the magnatude of the ftp process, ... you've hit the "bubble down" problem that I alluded to. ... When you're finished you can do a reverse alphabetical order delete ...
    (comp.os.vms)
  • Something better than ftp?
    ... We have some in house developed software that automatically updates a ... web server with static html pages by starting an ftp process using the ... system default ftp client. ... circumstances we observe the ftp process stalling occasionally (maybe 1 ...
    (uk.comp.os.linux)
  • Re: Something better than ftp?
    ... web server with static html pages by starting an ftp process using the ... system default ftp client. ... rsync. ... monitoring and the stalled ftp process killed. ...
    (uk.comp.os.linux)