Re: No of files in a directory (performance)

From: Bob Harris (harris_at_zk3.dec.com)
Date: 03/03/04

  • Next message: John R: "Re: User/Kernel Level Threads" 
    Date: Wed, 03 Mar 2004 16:03:26 GMT

    In article <ff81d3b0.0403030726.38e36a24@posting.google.com>,
     paltskallen@hotmail.com (Fredrik) wrote:

    > OK.
    >
    > I have noticed that rm -rf on a dir takes way longer on a dir with
    > many files in it, ex if there are 10k files it takes 10 sec, but with
    > 70k files it takes 150 sec.
    >
    > find|xargs rm
    > takes the same time (per file that is). It seems that -rf does some
    > magic that slows it down.
    >
    >
    > Bob Harris <harris@zk3.dec.com> wrote in message
    > news:<harris-8A4C82.20384627022004@cacnews.cac.cpqcorp.net>...
    > > In article <ff81d3b0.0402270129.205a3e@posting.google.com>,
    > > paltskallen@hotmail.com (Fredrik) wrote:
    > >
    > > > Hello!
    > > >
    > > > We have an application that creates and reads a lot of files.
    > > > These files are splitted on 100 dirs. Currently there are ~70 000 files
    > > > in
    > > > each dir.
    > > > Are there any limit that should be considered regarding to performance
    > > > issues??
    > > >
    > > > Using advfs and san disks.
    > >
    > > AdvFS in Tru64 UNIX v5.0 or later uses b-tree indexing for directory
    > > access. While a huge directory does take slightly longer to create,
    > > lookup, delete a file, it is _NOT_ linear. The performance is really
    > > very good.
    > >
    > > However, if you ask 'ls' to display all the files it will take a long
    > > time to fetch all the files, then sort them, and then display them, but
    > > if you use 'ls -f' it skips the sorting, which can make things faster.
    > >
    > > Bob Harris

    Did you do

        rm -rf *

    or

        rm -rf first_directory_to_remove

    If the first, then the shell you are using will first collect all the
    file names, then sort them, then substitute them on the command line,
    then it will invoke rm.

    The find command doesn't sort anything.

    If you did the 2nd example, then it must be something in rm, as the file
    system doesn't really know know what command is being used to delete
    something.

                                            Bob Harris

  • Next message: John R: "Re: User/Kernel Level Threads"

    Relevant Pages

    • coreutils-7.3 released [stable]
      ... been a few other interesting fixes and improvements. ... run a command like this: ... sort -m no longer segfaults when its output file is also an input file. ... destination file system with a fundamental block size of 4KiB or greater. ...
      (gnu.announce)
    • Re: FC1 Install _appeared_ to work, BUT...
      ... what sort of messages do make it to screen? ... I used to run RHL7.3, dual booting, on a Sony Vaio laptop. ... command prompt only safe-mode in windows but without the GUI). ...
      (Fedora)
    • Re: [OT] Compiler warning level
      ... > have command lines" when they enter the course. ... I started -- well, OK, unless you count the front panel switches and the ... lights on the panel could be called a GUI, they were sort of graphical ... I tend to write the makefile before typing in the start of the ...
      (comp.lang.c)
    • Re: getline caveats misuse howto
      ... populates the ERRNO builtin variable if getline fails. ... down the two-way pipe to the UNIX "sort" command. ... gawk terminates the coprocess and exits. ...
      (comp.lang.awk)
    • Re: BROWSE: order by column
      ... oHdr is now a reference to the header object ... descending sort. ... If you have a lot of columns that would be a lot of command ... Just add the code to change the sort order to a new ...
      (microsoft.public.fox.programmer.exchange)