Re: Panic on boot with April 16 src (lengthy info attached)

Robert Watson wrote:

On Tue, 17 Apr 2007, Robert Watson wrote:

I originally put it in there to work around a LOR that I was experiencing (based on you mentioning it in an email to current@ Sun 18 Mar 2007 15:50). http://sources.zabbadoz.net/freebsd/lor/191.html doesn't show any changes to that particular LOR, do you happen to know if there's any ongoing work on this? I'm very willing to act as a test system.

I chatted with Andre about the panic earlier this afternoon, and it sounds like the fix is straight forward. I would anticipate seeing it committed in the near future.

I'll send out an e-mail explaining the above lock order reversal tomorrow morning. I understand that several people have been looking at this, so perhaps one of those people will reply talking about it before then. :-)

The essential problem of this lock order reversal has to do with the fact that higher network stack layers hold locks over lower network stack layers. For example, the lock for a TCP connection is held over the operation to enqueue the TCP packet for transmission at a lower layer. This is necessary in order to maintain TCP transmission order into the transmission queue between multiple threads operating on the same TCP connection, as if the "transmit and enqueue" operation were non-atomic with respect to the same TCP connection in another thread, quite damaging reordering could take place. We directly dispatch the entire outbound network stack from that enqueue point, meaning that the per-TCP connection lock is held over that processing path, including the firewall. As a result, PCB locks (TCP connection locks) preceed the firewall in the lock order.

Firewall locks are about protecting the rule state of the firewall from corruption when firewall rules are updated, allowing readers to interpret the rules using a static snapshot, and writers to avoid mangling the rules via simultaneous non-atomic update. As such, when the firewall code is entered, the firewall lock is acquired, and held until the packet has been completely processed. Things get sticky deep in the firewall code because our firewalls include credential-aware rules, which essentially "peek up the stack" in order to decide what user is associated with a packet before delivery to the connection is done. The firewall rule lock is held over this lookup and inspection of TCP-layer state. In the out-bound path, we pass down the TCP state reference (PCB pointer) and guarantee the lock is already held. However, in the in-bound direction, the firewall has to do the full lookup and lock acquisition. Which reverses the lock order, and can lead to deadlocks.

I am doing work on fixing htis for ipfw.
it involves moving ipfw to a lockless method of operation.
(more info will be in the ipfw list in a few days)


debug.mpsafenet=0 places the Giant lock in front of all network stack lock acquisition, which effectively serializes all of the above. It doesn't remove the lock order reversal, but it does eliminate simultaneous lock acquisition, removing one of the necessary causes of deadlock. This trick of a serializing "global" lock in order to prevent lock order between "leaf" locks is not an uncommon technique, but in this case has a significant overhead (requiring non-parallelism in network processing), and needs to be fixed.

The key is to guarantee that the acquisition of the firewall reference will never be blocked waiting on a PCB lock -- i.e., that the firewall "lock" isn't a lock so much as a reference count that will never have to wait, removing the waiting requirement from the deadlock equation. I know that Julian Elischer has been looking at doing this, and others may have also. The model is essentially that you either starve writers to the firewall data, or you create a read-only snapshot to be used by readers in the event a writer arrives, allowing readers to pick up the new rules if available, or the old rules if not, and never wait indefinitely either way.

yep..
I have detailed plans afoot but not for pf.
I wouldn't know pf if it came up and kicked me in the shins so I'll be
leaving that to someone else.


Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"

_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: Zonealarm / Email
    ... > An IDS which runs on the same box as the spyware and even opens windows ... So for blocking already running malware on the box ID lock is no ... > for a security system if not counterproductive. ... A software firewall will not stop them, ...
    (comp.security.firewalls)
  • fixes for ipfw and pf lock ordering issues
    ... Currently, those who utilize ucred based firewalling, i.e. firewall ... rules which match based on UID, GID or JAIL ID are subject to lock order ... matching, namely ipfw and pf. ...
    (freebsd-hackers)
  • Re: Experiencing hangs on SMP box with no console messages given for clues. Details inside.
    ... the following lock messages came up -- I thought it might be related ... If using uid/gid firewall rules, make sure to read the pertinent man ... using a read-copy-replace write semantic with reference conts on the array.. ... previosu users ofthe structure just keep using the one they have ...
    (freebsd-current)
  • Re: Panic on boot with April 16 src (lengthy info attached)
    ... The essential problem of this lock order reversal has to do with the fact that higher network stack layers hold locks over lower network stack layers. ... This is necessary in order to maintain TCP transmission order into the transmission queue between multiple threads operating on the same TCP connection, as if the "transmit and enqueue" operation were non-atomic with respect to the same TCP connection in another thread, quite damaging reordering could take place. ... We directly dispatch the entire outbound network stack from that enqueue point, meaning that the per-TCP connection lock is held over that processing path, including the firewall. ...
    (freebsd-current)
  • Re: Server overloaded? Or is it a bug?
    ... then the kernel is at least a bit alive. ... Deadlock is a bit of a misnomer for what I have in mind. ... classes of things that look like deadlocks: ... Lock order problems are real deadlocks, where you grab locks in the wrong ...
    (freebsd-stable)