Fine-grained locking for UNIX domain sockets: patch updated


As part of Kris and Jeff's recent work on improving MySQL scalability on FreeBSD, I've updated my fine-grained locking patch for UNIX domain sockets to a more recent 7-CURRENT:

http://www.watson.org/~robert/freebsd/netperf/20070224-uds-fine-grain.diff

This patch replaces the global mutex with a global rwlock, and introduces per-unpcb locks to protect the state of each socket at the UNIX domain socket layer. Depending on the protocol operation, either the per-unpcb mutex is sufficient, or global read or write locks may also be required. The role of the global lock is to protect the topology of the connections between sockets, protect vnode v_socket pointers, and to prevent lock order issues in the event the per-unpcb lock of more than one socket must be acquired at a time. For example, exclusive global locking is required during bind() and connect() to protect v_socket and change the topology of connected sockets, whereas recv() requires only the unpcb lock of the receiving socket. send() may require either just an unpcb lock or a global lock if send() is implicitly connecting (sendto()) or disconnecting (PRUS_EOF) a socket.

This is basically the same patch that I have been maintaining since 2005 and posted at intermittent intervals when there have been reports of negative performance due to high contention on the UNIX domain socket subsystem lock. I continue to look for performance feedback, especially from workloads that are heavy consumers of UNIX domain sockets (such as MySQL). My experience has been that it gives a small performance hit on workloads and in configurations without much contention/scalability, but a measurable performance benefit with anything more substantial. Decrease in lower end workloads and configurations is at least in part due to the overhead of some additional locking vs. the benefits of reduced contention. Something that I am particularly interested in looking for is little or no performance hit on UP. This is a fairly heavily tested and refined patch, so the issue in deciding whether or not to commit this is its real-world effect on performance.

Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
freebsd-performance@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: Page fault in uipc_usrreq.c:997
    ... Kernel page fault with the following non-sleepable locks held: ... This is likely one of the remaining race conditions in UNIX domain sockets ... which may have "gone away" while the lock was ... but they are more exposed with greater kernel parallelism. ...
    (freebsd-current)
  • Fine-grained locking for UNIX domain sockets: patch updated
    ... This patch replaces the global mutex with a global rwlock, and introduces per-unpcb locks to protect the state of each socket at the UNIX domain socket layer. ... The role of the global lock is to protect the topology of the connections between sockets, protect vnode v_socket pointers, and to prevent lock order issues in the event the per-unpcb lock of more than one socket must be acquired at a time. ... For example, exclusive global locking is required during bindand connectto protect v_socket and change the topology of connected sockets, whereas recvrequires only the unpcb lock of the receiving socket. ... I continue to look for performance feedback, especially from workloads that are heavy consumers of UNIX domain sockets. ...
    (freebsd-current)
  • Re: HEADS UP: UNIX domain socket locking changes merged to CVS HEAD
    ... Please let me know if you experience any problems with UNIX domain sockets -- these changes will affect applications that consume UNIX domain sockets directly, like MySQL and Postfix, as well as consumers of POSIX fifos, which are implemented using UNIX domain sockets in-kernel. ... Lock: ... Pending Threads: ...
    (freebsd-current)
  • Re: Page fault in uipc_usrreq.c:997
    ... Kernel page fault with the following non-sleepable locks held: ... This is likely one of the remaining race conditions in UNIX domain sockets having to do with simultaneous connect and close, which occur due to dropping locks for either a blocking name lookup or a recursion via the socket layer into the protocol a second time. ... When the UNIX domain socket global lock is dropped and re-acquired, the UNIX domain socket code needs to re-evaluate its assumptions regarding any references it has to other UNIX domain sockets, which may have "gone away" while the lock was released. ... Interestingly, many of these races also existed in 4.x and before, but they are more exposed with greater kernel parallelism. ...
    (freebsd-current)
  • Re: [RFC/PATCH 1/8]: CGroup Files: Add locking mode to cgroups control files
    ... lockdep doesn't know anything about any of this. ... suitable lock types for each application, ... argument in favour of not having the locking done in the framework. ... with this patch it can't. ...
    (Linux-Kernel)