SUMMARY: /dev/dtremote symbolic link is breaking /dev/null

From: Greg Chavez (
Date: 02/15/05

  • Next message: "Need patch report for recommened patch cluster solaris 8 sparc DATED April 2004"
    Date: Tue, 15 Feb 2005 10:41:37 -0500

    Thank yous to Casper Dik and Crist Clark for providing me with some
    valuable investigative techniques. The cause of the incredible
    /dev/null metamorphosis turned out to be a configuration script for
    HP OpenView Operations 8.0 for Solaris. Specifically, the Microsoft
    Exchange "SPI" (smart plug-in) contained a configuration application,
    EXSPI Node Config (8.10), which was doing the deed. I am awaiting
    word from HP as concerns a patch.

    My suspicion that /dev/dtremote was somehow involved (it has a symb
    link to /dev/null) was debunked by both of my friendly advisers.
    Total red herring. I would often first notice the problem when
    someone would remote-CDE into the box. This requires a functional
    /dev/null. Jumped to conclusions. Oops.

    Generally, the mistake is a line of script which tries to use "mv" or
    "cp" to write to null, which of course does not work. The trick is
    revealing the culprit which, without the healing benefits of dtrace on
    Solaris 10, is beyond hard to do. If you run this command on any
    Solaris box, you will see what a daunting task it is to find out what
    script might cause /dev/null to be overwritten or deleted:

      # find / -type f | xargs file * | awk -F: '/script/ {print $1}' |
    xargs grep /dev/null

    And that's assuming it's not a binary. Only root can overwrite
    /dev/null, so what you are looking for is a script which runs as root.
     My situation was complicated by the fact that my client wanted to be
    able to login to CDE as root, widening the pool of suspects: cron,
    login scripts, /etc/dt CDE scripts. Any repository of local or
    third-party software should be scrutinized as well. More tips:

    Once you have determined that /dev/null has changed, run "ls -lc
    /dev/null". That will tell you the last time the permissions where

    Turn on process accounting (man acct). Concentrate on the processes
    that ran just before /dev/null changed. You can also look in
    /var/adm/messages for coinciding events. In my case, I wrote a script
    which checked /dev/null and, if missing or altered, sent a logger
    message to syslog with an "ls -lc" of /dev/null and recreated
    /dev/null. Keeps things working and helps you narrow down the time
    frame of the event.
    There are two ways to recreate /dev/null:

    # cd /dev
    # ln -s ../devices/psuedo/mm@0:null null
    [Make sure the link is made relative to /dev]

    # /usr/sbin/mknod /dev/null c 13 2
    # chown root:sys /dev/null ; chmod 666 /dev/null

    In the end, it was none of this, just a hunch, that lead us to
    discover the evil HP OVO program. Good times.

    --Greg Chavez
    ---------- Original message ----------
    From: Greg Chavez <>
    Date: Fri, 11 Feb 2005 09:30:32 -0500
    Subject: /dev/dtremote symbolic link is breaking /dev/null

    I am working short-term at a client site that relies heavily on remote
    CDE logins between their many Suns. The system on which I am working
    is a Sun Blade 2500 running Solaris 9, fully patched, JASS'd, and
    running HP OpenView Operations .. I built the OS myself. The gist is
    that /dev/null keeps getting turned into an empty file with 664
    permissions. Attempts to relink /dev/null to the null pseudo device
    last only until someone tries to perform a remote CDE login. The
    culprit appears to be /dev/dtremote which is symlinked to /dev/null.
    That would seem to be a problem to me, but this old post suggests

    As long as I change /dev/null from 644 to 666, this seems to cause no
    problems when the system is running. But this causes huge problems
    when the system is rebooted. Nothing can "create" /dev/null and the
    systems boots up in single-user, read only. I have to mount the OS
    from CD and relink /dev/null to the psuedo device to get it working
    again. Obviously, this is terrible.

    Does anybodh have any insight into this? If I can't figure it out,
    I'm going to advise my client to not remote-CDE into this box which,
    really, I should advise them to not do anyway. Nevertheless. They
    like things to work.

    Thanks, will sum.

    --Greg Chavez
    sunmanagers mailing list

  • Next message: "Need patch report for recommened patch cluster solaris 8 sparc DATED April 2004"