Re: Fine grain select locking.


On Thu, 2 Aug 2007, Alfred Perlstein wrote:

* Jeff Roberson <jroberson@xxxxxxxxxxxxxx> [070802 17:52] wrote:

I believe filedescriptor locking is the place where we are most lacking.
The new sx helped tremendously. However, this is still going to be a
scalability limiter. I have looked into both linux and solaris's solution
to this problem. Briefly, linux uses RCU to protect the list, which is
close to ideal as this is certainly a read heavy workload. Solaris on the
other hand uses the actual file lock to protect the descriptor slot. So
they fetch the file pointer, lock it, and then check to see if they lost a
race with the slot being reassigned while they were acquiring the lock.
This approach is perhaps better than rcu in many cases except when the
descriptor set is expanded. Then they have to lock every file in the set.

Certainly this is an extreme edge case... ?

Well that may be true, yes. However, there are other problems with this scheme. For example, flags settings could be done entirely with cmpset, without using a lock at all. In most cases we're just setting a bit which can be done with atomic_set. When we're doing multiple operations we could compute the value and attempt to est it in a loop. So we can totally eliminate locking the descriptor here.

We also could use atomic ops to protect the file descriptor reference count. This would eliminate another use of the FILE_LOCK(). I'm not sure if it's possible to merge this with an approach that uses the FILE_LOCK() to protect the descriptor table. Although I've not thought it all the way through.

If the ref count and flags were done with atomics the main consumer of FILE_LOCK would actually be the unix domain socket garbage collection code. How's that for old unix baggage. Do many programs actually pass around descriptors these days? inetd? others? It might be worth it to lock this seperately from the file lock.

Anyway, these things need to be explored for 8.0. With more cores and more multi-threaded applications file desc locking is one of the first points of contention as evidenced by the lock profiles and the sophistication of the solutions in other kernels.

Jeff



I could see it happening if we started with low limits, but perhaps
by keeping counters/stats we could tell people how to tune their
systems, or even autotune them.

-Alfred

_______________________________________________
freebsd-arch@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: LOR with filedesc structure and Giant
    ... We hold the file descriptor lock for the duration ... > because I need to grab a vnode lock to authorize polling the vnode using ... The easiest fix is to take a reference on the descriptors in the ...
    (freebsd-current)
  • Re: Fine grain select locking.
    ... Briefly, linux uses RCU to protect the list, which is ... other hand uses the actual file lock to protect the descriptor slot. ...
    (freebsd-arch)
  • Re: Multiple IMAP connections to same folder
    ... it has a shared lock open on the mailbox on the server. ... So it opens another descriptor append the message to the destination, keeping the first descriptor for the open mailbox. ...
    (comp.mail.imap)
  • Re: Debugging USB device
    ... as soon as I claim being to USB HID keypad (and of course ... Note that when I remove the "keyboard" part of the descriptor to ... keep only the "LED" part (CAPS LOCK, NUM LOCK, SCROLL LOCK), the device is ... The next plug will be again a first plug. ...
    (microsoft.public.development.device.drivers)
  • Re: Locking a picture or text box in a specific position
    ... You can lock the text box group by following these steps: ... you can protect the text box group as suggested in Jeffrey's mail. ... >> Anchor. ... >> select the picture.] ...
    (microsoft.public.mac.office.word)