Too many open files simultaneously

From: Roland Lenz (r.lenz_at_glastroesch.ch)
Date: 12/22/03 

Date: 22 Dec 2003 01:41:11 -0800

We have a problem with our production systems running Tru64 V.5.1.
Occasionaly our Applications (written in Cobol with MF Server Express
V2.2)
get an run-time error 9/014 (too many open files).

I used the lsof-tool (list open files) to show what happens. We are
running out of file-descriptors!

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
a131 486643 asc519 cwd VDIR 2765,89447 8192 103
/usr1/asc519
a131 486643 asc519 txt VREG 2765,89447 3211408 76345
/usr1/ps/tros/bin/a131
a131 486643 asc519 txt VREG 4053,353269 548256 1551
/usr/shlib/libcurses.so
a131 486643 asc519 txt VREG 4053,353269 679360 1538
/usr/shlib/ev6/libm.so
a131 486643 asc519 txt VREG 3309,565557 537536 2370
/opt/lib/cobol/lib/libcobcrtn.so.2
a131 486643 asc519 txt VREG 3309,565557 164064 2364
/opt/lib/cobol/lib/libcobmisc.so.2
a131 486643 asc519 txt VREG 3309,565557 41600 835
/shlib/libexc.so
a131 486643 asc519 txt VREG 3309,565557 2283904 3494
/shlib/libc.so
a131 486643 asc519 txt VREG 3309,565557 79424 2372
/opt/lib/cobol/lib/libcobscreen.so.2
a131 486643 asc519 txt VREG 3309,565557 817808 2347
/opt/lib/cobol/lib/cobmffh.so.2
a131 486643 asc519 txt VREG 3309,565557 70384 2340
/opt/lib/cobol/lib/cobmfini.so.2
a131 486643 asc519 txt VREG 3309,565557 1012720 2358
/opt/lib/cobol/lib/libcobrts.so.2
a131 486643 asc519 txt VREG 3309,565557 159424 3496
/sbin/loader
a131 486643 asc519 0r VCHR 2,2 0t0 27
/dev/null
a131 486643 asc519 1 can't
read file struct from 0xc00000ee009091a2
a131 486643 asc519 2w VREG 2765,89447 0 176177 /usr1
(usr1_domain#usr1)
a131 486643 asc519 3 can't
read file struct from 0xc00000ee009091a2
a131 486643 asc519 4w VREG 2765,89447 0 176177 /usr1
(usr1_domain#usr1)
a131 486643 asc519 5 can't
read file struct from 0xc00000ee009091a2
a131 486643 asc519 6r VCHR 2,2 0t0 27
/dev/null
a131 486643 asc519 7 can't
read file struct from 0xc00000ee009091a2
a131 486643 asc519 8r VCHR 2,2 0t0 27
/dev/null
a131 486643 asc519 9 can't
read file struct from 0xc00000ee009091a2
a131 486643 asc519 10r VCHR 2,2 0t0 27
/dev/null
a131 486643 asc519 11 can't
read file struct from 0xc00000ee009091a2
a131 486643 asc519 12r VCHR 2,2 0t0 27
/dev/null
a131 486643 asc519 13 can't
read file struct from 0xc00000ee009091a2
a131 486643 asc519 14r VCHR 2,2 0t0 27
/dev/null
a131 486643 asc519 15 can't
read file struct from 0xc00000ee009091a2
a131 486643 asc519 16r VCHR 2,2 0t0 27
/dev/null

... it goes on until 4096 file-descriptors are used!

a131 486643 asc519 4091 can't
read file struct from 0xc00000ee009091a2
a131 486643 asc519 4092r VCHR 2,2 0t0 27
/dev/null
a131 486643 asc519 4093 can't
read file struct from 0xc00000ee009091a2
a131 486643 asc519 4094r VCHR 2,2 0t0 27
/dev/null
a131 486643 asc519 4095 can't
read file struct from 0xc00000ee009091a2

Has anybody an idea?

Regards Roland