Re: Fine grain select locking.

Jeff Roberson wrote:

On Thu, 2 Aug 2007, Alfred Perlstein wrote:

* Jeff Roberson <jroberson@xxxxxxxxxxxxxx> [070802 17:52] wrote:

I believe filedescriptor locking is the place where we are most lacking.
The new sx helped tremendously. However, this is still going to be a
scalability limiter. I have looked into both linux and solaris's solution
to this problem. Briefly, linux uses RCU to protect the list, which is
close to ideal as this is certainly a read heavy workload. Solaris on the
other hand uses the actual file lock to protect the descriptor slot. So
they fetch the file pointer, lock it, and then check to see if they lost a
race with the slot being reassigned while they were acquiring the lock.
This approach is perhaps better than rcu in many cases except when the
descriptor set is expanded. Then they have to lock every file in the set.

Certainly this is an extreme edge case... ?

Well that may be true, yes. However, there are other problems with this scheme. For example, flags settings could be done entirely with cmpset, without using a lock at all. In most cases we're just setting a bit which can be done with atomic_set. When we're doing multiple operations we could compute the value and attempt to est it in a loop. So we can totally eliminate locking the descriptor here.

We also could use atomic ops to protect the file descriptor reference count. This would eliminate another use of the FILE_LOCK(). I'm not sure if it's possible to merge this with an approach that uses the FILE_LOCK() to protect the descriptor table. Although I've not thought it all the way through.

If the ref count and flags were done with atomics the main consumer of FILE_LOCK would actually be the unix domain socket garbage collection code. How's that for old unix baggage. Do many programs actually pass around descriptors these days? inetd? others? It might be worth it to lock this seperately from the file lock.

I'm sure I've alredy implemented it, but later I realized that there is a race with the p_fd field (if I got you right you are referring to the fdesc_mtx here), so we probabilly should better arrange those paths firstly.

Thanks,
Attilio


_______________________________________________
freebsd-arch@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: Fine grain select locking.
    ... other hand uses the actual file lock to protect the descriptor slot. ... If the ref count and flags were done with atomics the main consumer of FILE_LOCK would actually be the unix domain socket garbage collection code. ...
    (freebsd-arch)
  • Re: RT patch acceptance
    ... > Most spinlocks only protect small code parts. ... > larger codes can probably use optionally some different lock. ... So in that sense, if you do hard RT in the Linux kernel, it surely ...
    (Linux-Kernel)
  • Re: [PATCH 2.6.28-rc5 01/11] kmemleak: Add the base support
    ... It traces the memory allocation/freeing in a way similar to ... For some RCU implementations, but only by accident. ... search tree and removed in delete_objectwith this lock held ... OK, holding an object's lock can protect that object from deletion, ...
    (Linux-Kernel)
  • Re: [PATCH 2.6.28-rc5 01/11] kmemleak: Add the base support
    ... It traces the memory allocation/freeing in a way similar to ... search tree and removed in delete_objectwith this lock held ... OK, holding an object's lock can protect that object from deletion, ...
    (Linux-Kernel)
  • Re: [PATCH 2.6.28-rc5 01/11] kmemleak: Add the base support
    ... It traces the memory allocation/freeing in a way similar to ... search tree and removed in delete_objectwith this lock held ... Freeing a memleak_object is done via an RCU callback invoked from ... What exactly does object_tree_lock protect? ...
    (Linux-Kernel)