Re: How to enter DDB through a terminal server / remote console ... ?


On Jun 28, 2006, at 6:28 PM, Robert Watson wrote:


On Wed, 28 Jun 2006, User Freebsd wrote:

'k, now that I'm up to 3 6-STABLE servers that are deadlocking, I'm spending time with the remote tech today to get a serial console put online ... how do I drop into DDB remotely, where the serial console is going through a Portmaster Terminal server? issuing CTL-ALT-ESC, I doubt, will work, will it?

If configured to use a serial console (console="comconsole" in loader.conf), you can enter the debugger with BREAK_TO_DEBUGGER in the kernel config by sending a serial break. With my portmasters, I telnet to a TCP port to connect to the serial console, so I send a telnet break, using ^]send break.

I *used* to set BREAK_TO_DEBUGGER also, but I discovered (the hard, painful, revenue-costing way) that if you ever power cycle a Cyclades TS1000 terminal server, it sends a break signal down each and every serial line attached to it.

So, now I configure my debuggering in the kernel like this:

# Kernel debugging
options KDB #Enable the kernel debugger
options KDB_TRACE
options KDB_UNATTENDED
# Enable the ddb debugger backend.
options DDB
# break on sequence CR ~ ^b
options ALT_BREAK_TO_DEBUGGER


This lets me get the best of both worlds. It is amazing that the CR ~ ^b sequence will respond even when the box itself is totally unresponsive due to some runaway process or other shortage of memory condition.

Another thing you may wish to consider is adding option SW_WATCHDOG then running the watchdog daemon. It may help you by panicking your way out of deadlock automatically. You could also investigate getting a hardware watchdog card that works with watchdogd.

Relevant Pages

  • RE: Machine hangs(Beta7), only reset button works
    ... Ok, I managed to break into the with a break over serial console, attached ... > The box completely freeze typically when it's done running the first ... it's often possible to break into the debugger using a break over serial ... watchdog drivers -- some hardware comes with built in watchdog parts, ...
    (freebsd-current)
  • Re: How to troubleshoot solid freeze-up?
    ... > multiple identical NICs. ... The normal method is to use a break signal to get into the debugger. ... Serial break on the first serial console port. ... serial port can't interrupt the CPU to drop into the debugger. ...
    (freebsd-hackers)
  • Re: reboot without any info
    ... > moment I run totem in the debugger. ... I would suggest hooking up a serial or firewire console on this machine ... logged at a serial console. ...
    (freebsd-current)
  • Re: netstat fails with memory allocation error and error in kvm_read
    ... >>We just had another hard hang, and this time I managed to break to the ... >>debugger on the serial console. ... I got this stack trace ...
    (freebsd-stable)
  • Re: debugging kernel options
    ... is there any performance penalty of running such kernel? ... Right now these are sysctls, not tunables, but you can change the default using options KDB_UNATTENDED (which flips the default to not entering the debugger and rebooting) and options KDB_TRACE. ... In particular, it seems to focus mostly on crash dumps, and many problems are more easily debugged using information in DDB. ... I normally also use a serial console when debugging desktop boxes: I can always plug my notebook in with a serial cable to see why it's entered the debugger. ...
    (freebsd-stable)