Re: Samba limited to 32767 dirs?

From: Duh Duh (keepitall_at_1spam.biz)
Date: 03/13/05 

Date: Sun, 13 Mar 2005 03:21:08 GMT

<pakrat@localhost.private.neotoma.org> wrote in message
news:slrnd34c8c.mh3.pakrat@mouse.private.neotoma.org...
> On Fri, 11 Mar 2005 15:01:39 GMT in
> <npiYd.112140$pc5.59172@tornado.tampabay.rr.com> Duh Duh
> <keepitall@1spam.biz> wrote:
>> That is the problem. This is a user who has created a directory for each
>> letter of the alphabet, a-z and 0-9 to store domains in. Then under each
>> letter he has created each domain name in a seperate directory. These
>> are
>> the problem. He has more than 32767 directories that are domains that
>> begin
>> with w.... I am sure he will hit that limit in others as well.
>
> Would you mind explaining to me why a filesystem is being used for this
> instead of a database?

Database vendors tend to not handle this paricular system very well. The
application development group used 5 different database systems before, and
the cost was astroniomical in terms of support and license, etc.

They went to a simple flat file system to prevent those types of costs.
There are literally dozens of machines that read / write this data.
Following the unix tenant of the simplest design works in this case. Each
directory holds information that is required for that domain, which varies
depending upon the domain. And no app has to have a database license to
access the data. The data in question is around 500GB of total data. When
it was in a database it was around 2 TB's of data, DB's not being the most
efficient at storing data, but accessing it.

> In theory samba serving a share that is backed by a reiserfs filesystem
> on Linux should be able to handle this.
> In practice, I recall a recent thread where this eventually causes
> enough hash collisions that life sucks, but it might work for a few
> more orders of magnitude.

So the reiserfs does not have this limitation? That is something I did not
know. I will have to look into it.

>>
>> BTW, the directory is required because x number of pages is stored per
>> domain. Then that information is used by other apps who know the
>> directory
>> structure as well... If I have to tell him it is an OS limit that means
>> problems for me. I am the one who recommended we go to FreeBSD... :(
>> The
>> windows server they were using handled it just fine, but the drives were
>> always so fragmented that I could not even get a dir listing half the
>> time.
>> It was a constant nightmare of defragmenting.... yuck.
>>
>> Changing the algorithm would require groups of people changing their
>> code,
>> not something I want to recommend lightly. :)
>
> This sounds like the wrong design for the task at hand.
> At the very least use some sort of hashing alg to do the split
> instead of alphabetical.

That was used originally. That caused the generation of around 50,000
directories (buckets) that then a small number of sub-buckets (domains)
would live under. Of course this doesn't work well either...

I can suggest to them they change it to a more tiered structure, but once
again... This worked under windows. So I am going to have an uphill battle
telling them to change probably 40+ applications and systems that publish
data off this to clients just because bsd can't do what a windows server
did... Of course the windows server licenses were EXPENSIVE, and the
constant framentation of the drive is what was killing me (not that they
care). I was hoping to re-use that budget for the windows license to get 3
servers that could handle the load under bsd. I might be able to make a
case for it due to fault tolerance, have them change the algorithm to write
to more than one server at a time, so if a single server goes down we can
still do our jobs... I might try it.

Thanks.

Relevant Pages

  • Re: MySQL Database with FrontPage 2002
    ... I'm definitely referring to a Windows server. ... FrontPage creates ASP pages to work with databases, ... If the MySql database is hosted on a Windows server, ... The DBW works just fine with MySQL. ...
    (microsoft.public.frontpage.client)
  • Re: Membership Database
    ... ASPemail and ASPupload installed as well with mySQL, PHP, PERL and God ... I am going to try Thomas' solution at the MS link example ASP. ... (Requires Windows Server with support for ASP ... This shows how to control access to pages from a database of users, ...
    (microsoft.public.frontpage.programming)
  • Re: ntfrs error - corrupt ntfrs jet database
    ... Configuration and operational recommendations for the File Replication ... service in Windows Server 2003 and Windows 2000 Server ... Following the directions and deleting the database ...
    (microsoft.public.windows.server.networking)
  • Problem with Office Access 2003 on Windows Server 2003 Terminal Services
    ... I'm having a problem with Office Access 2003 running on Windows Server ... We run a Microsoft Access Database from our main file server which is ... The shortcut isn't the problem either. ... shortcut and the database itself allows access to all users and admins. ...
    (microsoft.public.windows.server.general)
  • Re: Append query to multiple tables
    ... the database to give the actual table and field names). ... > values for the foreign key fields in the License table?" ... > you are adding the new record (appending the record). ... > primary key value that is created when the record is being added ...
    (microsoft.public.access.externaldata)
Loading