Embedding a vnode type to its interlock mutex type

• Previous message: Pedro F. Giffuni: "Re: Loadable VM pager"
• Next in thread: Bruce M Simpson: "Re: Embedding a vnode type to its interlock mutex type"
• Reply: Bruce M Simpson: "Re: Embedding a vnode type to its interlock mutex type"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Date: Mon, 18 Aug 2003 16:13:26 +0900
To: arch@freebsd.org

In short: A vnode should embed its type name (VREG, VCHR, etc.) in the
type of the interlock mutex to avoid a false LOR alarm by Witness.

The Details:

With my p4 branch (tanimura_socket), I get a LOR shown below:

--- v --- LOR log --- v ---
lock order reversal
 1st 0xfffff800262faa90 local rcv (local rcv) @ kern/uipc_socket.c:1250
 2nd 0xc03c9a08 vm page queue mutex (vm page queue mutex) @ kern/uipc_cow.c:88
Stack backtrace:
_mtx_lock_flags() at _mtx_lock_flags+0x9c
socow_iodone() at socow_iodone+0x1c
m_free() at m_free+0xf4
soreceive() at soreceive+0xe50
soo_read() at soo_read+0x78
dofileread() at dofileread+0x60
read() at read+0x3c
syscall() at syscall+0x280
--- ^ --- LOR log --- ^ ---

where the lock order is:

--- v --- lock order by Witness --- v ---
4 pseudofs_vncache -- last acquired @ fs/pseudofs/pseudofs_vncache.c:227
5 mntvnode -- last acquired @ nfsclient/nfs_vfsops.c:1014
13 vnode interlock -- last acquired @ nfsclient/nfs_vfsops.c:1010
14 Syncer mtx -- last acquired @ kern/vfs_subr.c:1798
14 spechash -- last acquired @ kern/vfs_subr.c:2009
14 vnode_free_list -- last acquired @ kern/vfs_subr.c:760
14 local snd -- last acquired @ kern/uipc_socket.c:2173
15 local rcv -- last acquired @ kern/uipc_socket.c:2174
16 mbuf PCPU list lock -- last acquired @ kern/subr_mbuf.c:926
17 mbuf subsystem general lists lock -- last acquired @ kern/subr_mbuf.c:676
16 Malloc Stats -- last acquired @ kern/kern_malloc.c:324
16 sleep mtxpool -- last acquired @ kern/kern_prot.c:1686
16 socket generation -- last acquired @ kern/uipc_socket.c:263
16 sellck -- last acquired @ kern/sys_generic.c:816
16 UMA pcpu -- (already displayed)
10 vm object_list -- last acquired @ vm/vm_object.c:621
11 kmem object -- last acquired @ vm/vm_meter.c:179
12 vm page queue mutex -- last acquired @ vm/vm_pageout.c:1464
13 vnode interlock -- (already displayed)
--- ^ --- lock order by Witness --- ^ ---

Witness yells out because:

- vm page queue mutex is immediately followed by vnode interlock, and

- local rcv (the receive lock of an AF_LOCAL socket) is locked after
  vnode interlock in another place.

Witness seems to thus presume that vm page queue mutex should have
been acquired before local rcv. Although Witness treats this as a
LOR, I believe this should be safe because AF_LOCAL sockets and VSOCK
vnodes should not be used during vm operation.

If we distinguished the interlocks for VSOCK vnodes from the ones for
VREG (or VCHR), then there would be two lock orders:

vm page queue mutex
 VREG/VCHR vnode interlock
  Syncer mtx
  spechash
   :
   :

and

VSOCK vnode interlock
 local snd
  local rcv
   vm page queue mutex
    :
    :

They should safely coexist with each other.

In order to accomplish that, the type of a mutex must be changed when
the type of a vnode is changed. While it might work to just destroy
and reinit the interlock of a vnode, those operations are likely to be
overdoing because the type of a mutex is meaningful for only Witness.
It would hence be better for Witness to provide an API to change the
type of an inited mutex.

For sockets, I implemented a trick to change lock types quite a few
days ago. The idea is to embed the address family name to the types
of socket locks. It solved almost all of the false LOR alerts for
routing sockets, where a routing socket may need to be altered during
sending from an inet or inet6 socket. It is quite easy for sockets to
change its lock type because a socket is always freed when it is
closed.

Comments?

-- 
Seigo Tanimura <tanimura@tanimura.dyndns.org> <tanimura@FreeBSD.org>
_______________________________________________
freebsd-arch@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"

• Previous message: Pedro F. Giffuni: "Re: Loadable VM pager"
• Next in thread: Bruce M Simpson: "Re: Embedding a vnode type to its interlock mutex type"
• Reply: Bruce M Simpson: "Re: Embedding a vnode type to its interlock mutex type"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]