Re: Do too many files hurt a directory?

From: Charlie Gibbs (cgibbs_at_kltpzyxm.invalid)
Date: 03/26/04 

Date: 26 Mar 04 13:37:45 -0800

In article <2S_8c.22903$tY6.670359@bgtnsc04-news.ops.worldnet.att.net>
hukolau@NOSPAM.att.net (Nick Landsberg) writes:

>Lew Pitcher wrote:
>
>> Charlie Gibbs wrote:
>>
>>>A process went wild at a customer site the other night, and started
>>>creating files with gay abandon. When the staff came in the next
>>>morning, the system was extremely sluggish. I dialed in and found
>>>that over 280,000 files had been created, all in the same directory.
>>>All of the files belong to root (the user kicked off the process's
>>>parent under the wrong ID), and most of them are empty or nearly so.
>>>I finally managed to delete all of the files (it took the better part
>>>of a day), but the system is still sluggish - even a simple ls
>>>command takes 10 seconds or more to respond.
>>
>> Depending on the filesystem in question, filename location /may/ be
>> implemented as a linear search of the directory. In this case, I'd
>> expect that the system would be very sluggish whenever the overloaded
>> directory is read.
>>
>> However, I thought that 'modern' filesystems implemented various
>> directory search improvements that eliminated the linear search.
>
>I did too (think that modern filesystems were smarter).

Ah, but the key word here is "modern". They're running SCO 5.0.5.

>In any event, the directory itself is also a special file and
>once it is "stretched out of shape" the lookup can get costly
>in terms of disk I/O's. Assuming a "short" filenames of
>10 chars, that's 2.8 MB of space that it's using up
>and (on average) 1.4 MB of stuff to look up if a linear search
>is being used.

I suspected something like that was happening.

>The most expensive operation would be to look for a non-existant
>file, too.

That makes sense too, and indeed was happening.

>>>Is a directory somehow stretched out of shape if it has an absurdly
>>>high number of files created? If so, what's the best way to recover?
>>
>> Probably by moving the remaining files out of the current directory
>> and into a new directory. Once that's done, remove the old directory
>> completely, and move the new directory into it's place. This should
>> improve things by re-optimizing the directory search.
>>
>>>Currently I have the customer creating a new directory and copying
>>>the contents of the old directory into it. Then she can delete the
>>>old directory and rename the new one.
>>
>> See above. We seem to think alike <grin>
>
>Yep, that's the ticket! For future reference,
>cpio can be used quite effectively for the
>copy by actually creating links rather than
>copies.
>
>mkdir newdir
>cd olddir
>find . -print | cpio -pludmav ../newdir
>(this preserves the access and modification times,
>if that's important to you. The "l" option
>only hard links the files into ../newdir
>rather than actually copying them)
>cd ..
>rm -rf olddir
>mv newdir olddir

I'll tuck away that cpio trick for next time. The customer did
it the good old-fashioned way with cp, and it was slooooooow.

>>>We've got our fingers crossed - the way the system is running
>>>it's going to take a long time - but if it fails, what then?
>>>Can fsck clean things up, or is it time for a full reformat
>>>a la Windows?
>>
>> Gak! You Always Have Other Options. We'll think of something.

:-) Actually, copying everything into a new directory did the
trick. The system is now as snappy as it ever was. Many thanks
to you all.

As an interesting postscript, when I dialed back into the system
I was still logged in (I've already pointed out to them what a
security hole it is to not log someone out when the connection
drops), and it was very strange to find myself in a directory
that had been unlinked. Most commands died immediately, complaining
about an invalid directory, and pwd returned a completely blank line.
When I logged out and back in again, everything was back to normal. 

--
/~\  cgibbs@kltpzyxm.invalid (Charlie Gibbs)
\ /  I'm really at ac.dekanfrus if you read it the right way.
 X   Top-posted messages will probably be ignored.  See RFC1855.
/ \  HTML will DEFINITELY be ignored.  Join the ASCII ribbon campaign!