large phantom files

We've had 2 incidents this week where file systems were 100% or quite
close.

In both scenarios du was reporting little to no utilization. So, I
figured it was the old issue where I just needed to identify the
application that still had a file descriptor open on some really large
file and shut the app down to clear the space. Unfortunately, that
didn't prove to be the case.



The first incident was /tmp on an apache server. After much hunting
around, the conclusion was that the application (a weblogic agent) was
writing something to /tmp and someone deleted the file. I suppose the
possibilities are limitless. In the end, after stopping & starting
several of the 30+ apache instances, we ended up rebooting.

Fortunately it was a dev environment, but I still took enough heat.



2nd incident was a less critical orca server, /var was showing 98% full
but only had 25MB worth of files out there.

Orca server was sol10 but at end of day so we went with the easy fix &
rebooted.



I'm looking for other strategies, tools, whatever to be able to locate
the offending process in this scenario. On the web server, I exhausted a
lot of time with this looking at /proc & pfiles command & lsof, etc.
This was solaris 9



Also curious if dtrace could've helped in any way.



Joe Beck Ciber Inc. - a consultant to SEI One Freedom Valley Drive/ 100
Cider Mill Road| Oaks, PA 19456 | p: 610.676.2258 | jbeck@xxxxxxxx
_______________________________________________
sunmanagers mailing list
sunmanagers@xxxxxxxxxxxxxxx
http://www.sunmanagers.org/mailman/listinfo/sunmanagers