No more processes - what went wrong?

From: Dave (nospam_at_nowhere.com)
Date: 02/26/05

  • Next message: Richard B. Gilbert: "Re: StorEdge T3: lost password" 
    Date: Sat, 26 Feb 2005 14:19:46 +0000

    I have a quad processor Ultra 80 with 4GB RAM running Solaris 9.

    sparrow /export/home/drkirkby % uname -a
    SunOS sparrow 5.9 Generic_117171-15 sun4u sparc SUNW,Ultra-80

    The day before yesterday it was behaving odd. I was using Matlab (which
    I don't do that often), and it was not responding very well. Later
    Solaris was saying 'No more processes' if I tried to run a command. I
    shut down Matlab, the error went away, and I thought no more of it.

    Yesterday morning this occurred again, although Matlab was not involved.
    Below is the output from ps -elf. As you can see, there are not than
    many processes reported. At the very end you will my attempt to count them

    sparrow /export/home/drkirkby % ps -elf | wc -l
    No more processes.

    but that fails as you see. I had to go to work, so left the machine (its
    a home machine), but managed to email the output of ps -elf to work
    before doing so.

    Then I tried to ssh in from work to the home machine and was unable to.
    Trying as root it always said incorrect password. Root logins are
    allowed (a hardware firewall limits ssh to only half a dozen IP's), but
    ssh kept reporting the password was wrong. Trying as a normal user, it
    would connect, report a message about 'no unauthorised access' which I
    have in /etc/motd, but then hang with an 'out of memory' message. I
    could never get as far as seeing a shell prompt.

    I spoke to one of the UNIX system admins at work (I did not show him the
    ps output), who said he had seen this before, and it can happen if /tmp
    gets too full (I guess he meant the 'out of memory' message). He said
    the best solution was to reboot.

    I don't trust the memory stats from 'top' as I know Solaris does not
    reclaim memory unless it needs it, but I must admit normally the 'free
    amount' bears some relationship to what I would expect. i.e. if the
    machine is doing nothing, I expect it to be around 3.5GB free, and if a
    program allocates a huge amount of RAM, I see this fall. When I came
    home last night, the memory reported by 'top' was only a few tens of MB,
    which is unusually low on the machine with 4GB RAM.

    I did what the UNIX admin at work suggested (reboot) and it seems OK.
    But any idea what can cause this? Goggleing around I see others
    reporting this, but nothing recent. I don't suppose what was relavent in
    the era of Solaris 2.5 is on Solaris 9.

    Here's the output from ps -elf, on the odd occasion I managed to
    actually get Solaris to respond. sensibly. There seems a lot of
    instances of /bin/tcsh. That is possibly OK - I tend to open lots of
    windows. The httpd processes running was not intensional, but I realsied
    these were being started at boot. I've just stopped that, although there
    was no security risk, as this is behind a hardware firewall. Does anyone
    see anything else odd?

    sparrow /export/home/drkirkby % ps -elf
    F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY
          TIME CMD
    19 T root 0 0 0 0 SY ? 0 Jan 30 ?
      0:00 sched
    8 S root 1 0 0 40 20 ? 158 ? Jan 30 ?
           0:33 /etc/init -
    19 S root 2 0 0 0 SY ? 0 ? Jan 30 ?
            0:00 pageout
    19 S root 3 0 1 0 SY ? 0 ? Jan 30 ?
           335:49 fsflush
    8 S drkirkby 16926 861 0 50 20 ? 399 ? Feb 21 ?
           0:00 /usr/dt/bin/dtexec -open 0 -ttproci
    8 S drkirkby 554 460 0 40 20 ? 25888 ? Jan 30 ?
          503:15 /usr/openwin/bin/Xsun :0 -nobanner
    8 S root 779 1 0 40 20 ? 254 ? Jan 30 ?
           0:00 /opt/SUNWspci2/bin/sunpcid
    8 S root 64 1 0 40 20 ? 295 ? Jan 30 ?
           0:00 /usr/lib/sysevent/syseventd
    8 S root 74 1 0 40 20 ? 355 ? Jan 30 ?
           0:00 /usr/lib/picl/picld
    8 S root 552 1 0 0 RT ? 268 ? Jan 30 ?
           0:00 /usr/lib/inet/xntpd
    8 S root 219 1 0 40 20 ? 277 ? Jan 30 ?
           0:00 /usr/sbin/rpcbind
    8 S root 469 1 0 40 20 ? 345 ? Jan 30 ?
           0:23 /usr/local/samba/bin/nmbd -D -l /va
    8 S root 555 460 0 40 20 ? 874 ? Jan 30 ?
           0:00 /usr/dt/bin/dtlogin -daemon
    8 S root 498 1 0 40 20 ? 327 ? Jan 30 ?
           0:00 /usr/lib/nfs/mountd
    8 S root 755 1 0 40 20 ? 405 ? Jan 30 ?
           0:00 ipmon
    8 S root 250 1 0 40 20 ? 470 ? Jan 30 ?
           0:49 /usr/sbin/syslogd
    8 S root 257 1 0 40 20 ? 288 ? Jan 30 ?
           0:08 /usr/sbin/cron
    8 S drkirkby 5109 5107 0 50 20 ? 324 ? 10:29:02 ?
           0:00 /var/tmp/lm_TMW.ld -z -c /var/tmp/l
    8 S root 239 1 0 40 20 ? 294 ? Jan 30 ?
           0:00 /usr/sbin/inetd -s
    8 S root 786 1 0 40 20 ? 337 ? Jan 30 ?
           0:00 /usr/lib/ssh/sshd
    8 S root 272 1 0 40 20 ? 448 ? Jan 30 ?
           0:21 /usr/sbin/nscd
    8 S root 278 1 0 40 20 ? 397 ? Jan 30 ?
           0:00 /usr/lib/lpsched
    8 S smmsp 305 1 0 40 20 ? 547 ? Jan 30 ?
           0:00 /usr/lib/sendmail -Ac -q15m
    8 S root 782 1 0 40 20 ? 226 ? Jan 30
    pts/1 0:00 /usr/lib/saf/ttymon -g -h -p sparro
    8 S root 325 324 0 40 20 ? 220 ? Jan 30 ?
           0:00 /usr/sadm/lib/smc/bin/smcboot
    8 S drkirkby 5507 861 0 50 20 ? 399 ? 10:29:57 ?
           0:00 /usr/dt/bin/dtexec -open 0 -ttproci
    8 S root 312 1 0 40 20 ? 551 ? Jan 30 ?
           0:31 /usr/lib/sendmail -bd -q15m
    8 S root 311 1 0 40 20 ? 135 ? Jan 30 ?
           0:00 /usr/lib/utmpd
    8 S root 467 1 0 40 20 ? 482 ? Jan 30 ?
           0:00 /usr/local/samba/bin/smbd -D -l /va
    8 S root 556 460 0 40 20 ? 302 ? Jan 30 ??
           0:00 /usr/openwin/bin/fbconsole -d :0
    8 S root 324 1 0 40 20 ? 220 ? Jan 30 ?
           0:00 /usr/sadm/lib/smc/bin/smcboot
    8 S root 326 324 0 40 20 ? 220 ? Jan 30 ?
           0:00 /usr/sadm/lib/smc/bin/smcboot
    8 S root 500 1 0 40 20 ? 276 ? Jan 30 ?
           0:00 /usr/lib/nfs/nfsd
    8 S nobody 787 762 0 40 20 ? 768 ? Jan 30 ?
           0:00 /usr/apache/bin/httpd
    8 S drkirkby 5502 5500 0 50 20 ? 323 ? 10:29:49 ?
           0:00 /var/tmp/lm_TMW.ld -z -c /var/tmp/l
    8 S root 1058 1 0 0 RT ? 268 ? Jan 30 ?
           4:32 /usr/lib/inet/xntpd
    8 S root 460 1 0 40 20 ? 642 ? Jan 30 ?
           0:00 /usr/dt/bin/dtlogin -daemon
    8 S drkirkby 5107 1 0 50 20 ? 316 ? 10:29:02 ?
           0:00 tcsh -c umask 022; /var/tmp/lm_TMW.
    8 S drkirkby 796 555 0 40 20 ? 241 ? Jan 30 ?
           0:00 /bin/ksh /usr/dt/bin/Xsession
    8 S root 762 1 0 40 20 ? 757 ? Jan 30 ?
           0:00 /usr/apache/bin/httpd
    8 S nobody 788 762 0 40 20 ? 761 ? Jan 30 ?
           0:00 /usr/apache/bin/httpd
    8 S nobody 789 762 0 40 20 ? 761 ? Jan 30 ?
           0:00 /usr/apache/bin/httpd
    8 S nobody 790 762 0 40 20 ? 761 ? Jan 30 ?
           0:00 /usr/apache/bin/httpd
    8 S nobody 791 762 0 40 20 ? 761 ? Jan 30 ?
           0:00 /usr/apache/bin/httpd
    8 S drkirkby 806 796 0 40 20 ? 302 ? Jan 30 ?
           0:00 /usr/openwin/bin/fbconsole
    8 S drkirkby 862 854 0 50 20 ? 880 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dtRbaaR
    8 S drkirkby 854 843 0 50 20 ? 968 ? Jan 30
    pts/3 0:00 /usr/dt/bin/dtsession
    8 S drkirkby 810 1 0 40 20 ? 340 ? Jan 30 ?
           0:00 /usr/openwin/bin/speckeysd
    8 S drkirkby 843 840 0 40 20 ? 337 ? Jan 30
    pts/3 0:00 tcsh -c unsetenv _ PWD; source
    8 S drkirkby 840 796 0 40 20 ? 505 ? Jan 30
    pts/3 0:00 /usr/dt/bin/sdt_shell -c unsetenv _
    8 S drkirkby 841 1 0 40 20 ? 305 ? Jan 30 ?
           0:00 /usr/dt/bin/dsdm
    8 S root 855 239 0 40 20 ? 447 ? Jan 30 ?
           0:00 rpc.ttdbserverd
    8 S drkirkby 853 1 0 40 20 ? 745 ? Jan 30
    pts/3 0:02 /usr/dt/bin/ttsession
    8 S drkirkby 861 854 0 50 20 ? 1364 ? Jan 30 ?
           7:50 dtwm
    8 S drkirkby 864 854 0 50 20 ? 880 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dt_baiR
    8 S drkirkby 863 854 0 50 20 ? 900 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dt5baqR
    8 S drkirkby 865 854 0 50 20 ? 880 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dtccayR
    8 S drkirkby 866 854 0 50 20 ? 880 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dtkcaOR
    8 S drkirkby 867 854 0 50 20 ? 887 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dtsca4S
    8 S drkirkby 868 854 0 50 20 ? 888 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dtycaGR
    8 S drkirkby 869 854 0 50 20 ? 901 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dtPcaiS
    8 S drkirkby 870 854 0 50 20 ? 902 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dtXcaOS
    8 S drkirkby 871 854 0 50 20 ? 887 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dt1caqS
    8 S drkirkby 872 854 0 50 20 ? 880 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dt5ca4R
    8 S drkirkby 873 854 0 50 20 ? 894 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dt9caaS
    8 S drkirkby 874 854 0 50 20 ? 880 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dtcdaGS
    8 S drkirkby 875 854 0 50 20 ? 904 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dtfdayS
    8 S drkirkby 877 854 0 50 20 ? 895 ? Jan 30 ??
           0:00 /usr/dt/bin/dtterm -session dtmdaWR
    8 S drkirkby 9737 861 0 50 20 ? 399 ? 15:08:05 ?
           0:00 /usr/dt/bin/dtexec -open 0 -ttproci
    8 S drkirkby 889 862 0 50 20 ? 391 ? Jan 30
    pts/4 0:00 /bin/tcsh
    8 S drkirkby 880 1 0 40 20 ? 240 ? Jan 30 ?
           0:00 /bin/ksh /usr/dt/bin/sdtvolcheck -d
    8 S drkirkby 898 864 0 50 20 ? 392 ? Jan 30
    pts/8 0:00 /bin/tcsh
    8 S drkirkby 6855 880 0 40 20 ? 121 ? Feb 02 ?
           0:00 /bin/cat /tmp/.removable/notify880
    8 S drkirkby 890 863 0 50 20 ? 392 ? Jan 30
    pts/5 0:00 /bin/tcsh
    8 S drkirkby 896 866 0 50 20 ? 392 ? Jan 30
    pts/7 0:00 /bin/tcsh
    8 S drkirkby 899 877 0 50 20 ? 384 ? Jan 30
    pts/9 0:00 /bin/tcsh
    8 S drkirkby 911 872 0 50 20 ? 392 ? Jan 30
    pts/10 0:00 /bin/tcsh
    8 S drkirkby 10213 10212 0 50 20 ? 1045 ? 16:45:28 ?
           0:55 /usr/dt/bin/sdtaudio
    8 S drkirkby 903 865 0 50 20 ? 392 ? Jan 30
    pts/6 0:00 /bin/tcsh
    8 S drkirkby 916 871 0 50 20 ? 396 ? Jan 30
    pts/12 0:00 /bin/tcsh
    8 S drkirkby 939 869 0 50 20 ? 392 ? Jan 30
    pts/17 0:00 /bin/tcsh
    8 S drkirkby 915 867 0 50 20 ? 392 ? Jan 30
    pts/11 0:00 /bin/tcsh
    8 S drkirkby 922 873 0 50 20 ? 399 ? Jan 30
    pts/13 0:00 /bin/tcsh
    8 S drkirkby 931 870 0 50 20 ? 391 ? Jan 30
    pts/14 0:00 /bin/tcsh
    8 S drkirkby 948 875 0 50 20 ? 396 ? Jan 30
    pts/20 0:00 /bin/tcsh
    8 S drkirkby 935 874 0 50 20 ? 392 ? Jan 30
    pts/16 0:00 /bin/tcsh
    8 S drkirkby 10107 1 0 50 20 ? 6460 ? 16:20:09 ?
           0:26 /usr/local/stow/acrobat-5.07/Reader
    8 S root 955 1 0 40 20 ? 353 ? Jan 30 ?
           0:00 devfsadmd
    8 S drkirkby 951 868 0 50 20 ? 417 ? Jan 30
    pts/19 0:00 /bin/tcsh
    8 S drkirkby 10056 861 0 50 20 ? 399 ? 16:15:30 ?
           0:00 /usr/dt/bin/dtexec -open 0 -ttproci
    8 S drkirkby 18759 1 0 40 20 ? 362 ? Feb 16 ?
           0:00 /usr/bin/ssh -x -oFallBackToRsh no
    8 S drkirkby 4542 861 0 50 20 ? 399 ? Feb 19 ?
           0:00 /usr/dt/bin/dtexec -open 0 -ttproci
    8 S drkirkby 19468 1 0 50 20 ? 2059 ? Feb 03 ?
          52:51 /opt/SUNWspci2/bin/sunpcbinary
    8 S root 11109 810 0 40 0 ? 696 ? 20:18:57 ?
           0:00 /usr/openwin/bin/sys-suspend
    8 S drkirkby 12310 12267 0 50 20 ? 140 ? 00:04:39
    pts/15 0:00 /usr/local/Matlab-R13/bin/sol2/matl
    8 S drkirkby 16929 16927 0 50 20 ? 422 ? Feb 21
    pts/46 0:00 /bin/tcsh
    8 S drkirkby 9740 9738 0 50 20 ? 393 ? 15:08:06
    pts/21 0:00 /bin/tcsh
    8 S drkirkby 5508 5507 0 40 20 ? 880 ? 10:29:58 ??
           0:00 /usr/dt/bin/dtterm
    8 S nobody 20621 762 0 40 20 ? 763 ? Feb 01 ?
           0:00 /usr/apache/bin/httpd
    8 S drkirkby 6808 6807 0 40 20 ? 1076 ? Feb 02
    pts/3 0:00 dtfile -noview
    8 S drkirkby 6807 853 0 40 20 ? 137 ? Feb 02
    pts/3 0:00 /bin/sh -c dtfile -noview
    8 S drkirkby 16116 1 0 50 20 ? 362 ? Feb 20 ?
           0:01 /usr/bin/ssh -x -oFallBackToRsh no
    8 S drkirkby 5111 5109 0 50 20 ? 366 ? 10:29:02 ?
           0:01 MLM -T sparrow 8.0 4 -c /var/tmp/lm
    8 S drkirkby 4543 4542 0 50 20 ? 924 ? Feb 19 ?
           0:00 /usr/dt/bin/dtstyle
    8 S drkirkby 10059 10057 0 50 20 ? 393 ? 16:15:30
    pts/22 0:00 /bin/tcsh
    8 S drkirkby 10057 10056 0 50 20 ? 880 ? 16:15:30 ??
           0:00 /usr/dt/bin/dtterm
    8 S drkirkby 10212 861 0 50 20 ? 399 ? 16:45:28 ?
           0:00 /usr/dt/bin/dtexec -open 0 -ttproci
    8 S drkirkby 5500 1 0 50 20 ? 316 ? 10:29:49 ?
           0:00 tcsh -c umask 022; /var/tmp/lm_TMW.
    8 S drkirkby 9738 9737 0 50 20 ? 879 ? 15:08:05 ??
           0:00 /usr/dt/bin/dtterm
    8 O drkirkby 14180 5510 0 50 20 ? 144 09:01:49
    pts/32 0:00 ps -elf
    8 S drkirkby 5503 5500 0 50 20 ? 137 ? 10:29:49 ?
           0:00 sh -c while read line; do echo "$li
    8 S root 20838 1 0 0 RT ? 779 ? Feb 17 ?
           0:00 cdrecord dev= 0,6,0 gracetime=2 fs=
    8 S drkirkby 12267 10059 0 50 20 ? 25007 ? 00:04:37
    pts/22 3:41 /usr/local/Matlab-R13/bin/sol2/matl
    8 S drkirkby 5510 5508 0 40 20 ? 393 ? 10:29:58
    pts/32 0:00 /bin/tcsh
    8 S drkirkby 16927 16926 0 50 20 ? 907 ? Feb 21 ??
          0:01 /usr/dt/bin/dtterm
    sparrow /export/home/drkirkby % ps -elf | wc -l
    No more processes.
    sparrow /export/home/drkirkby %

  • Next message: Richard B. Gilbert: "Re: StorEdge T3: lost password"

    Relevant Pages

    • Re: Canadian OpenVMS Seminar (07.02.20)
      ... appropriate answer to the question of "What specific RAS-related features ... Re: Solaris x86: Neil wrote about "running Solaris or LINUX on x86 to see ... support for DDR2 memory and support for online spare memory chips, ... relationship with Intel)? ...
      (comp.os.vms)
    • Re: steps to handle system problems
      ... User reports of "The application is slow" is just noise unless you can pin down time of day, activities on the network or server running it. ... I/O and memory stats over the long term to look at reports of "slow" with time and activity. ... In a galaxy long ago and far far away, an instructor in system tuning told my class that you get the most bang for your buck by first tuning the application, then maybe 10% from tuning the system. ... She was talking about another platform and O/S entirely different from Sun Solaris. ...
      (comp.unix.solaris)
    • Re: r2008a Slow on Solaris
      ... I run Matlab on several different machines at my work. ... the Sun to Solaris 10 and have put r2008a onto it. ... I've noticed Mathematica seems to do really poorly on Solaris SPARC, for reasons I don't know, whereas on Solaris x86 Mathematica works very well. ... The reference machine is a 2.4 GHz Pentium which scored 1.00. ...
      (comp.soft-sys.matlab)
    • MATLAB dropping Solaris support
      ... Mathworks will be dropping support for Matlab on Solaris as ... Matworks rep says that "there aren't any near term plans" to support x86 ...
      (SunManagers)
    • Re: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY
      ... I'm curious to know how Solaris deals with these problems, ... default filesystem in OpenSolaris is now ZFS. ... The ARC is separate from the general VM cache in solaris, too, IIRC. ... Most of the problems where ZFS runs the machine out of kernel memory (or ...
      (freebsd-stable)