Pernicious problem with vfork / qmail / qmail-scanner

From: Justin Baugh, KSC (baughj_at_discordians.net)
Date: 03/12/04

  • Next message: Paul Seniura: "Re: I have some questions about telnet/telnetd/libtelnet/tn3270 and why FreeBSD is different than other BSDs in this regard" 
    Date: Thu, 11 Mar 2004 22:12:25 -0500
To: freebsd-questions@freebsd.org

    Hello list,

    I am having a hell of a time with qmail/qmail-scanner, and I am hoping
    someone on the list can help me. Normally this setup has worked
    completely fine on any FreeBSD machine I've set it up on.

    This is on 5.1-p10 (which I'm still running as I've yet to upgrade to
    5.2.1R, and I am wondering if that just might be the problem).

    I am running a large mailserver which feeds the message to
    qmail-scanner, and then to spamd & a virus scanner (Kaspersky). This is
    a very beefy box (dual 2.4ghz Xeon, 2gb RAM) which should be able to
    handle many, many connections.

    My connection limits on tcpserver are set to 60 concurrent connections.
    concurrencyincoming/remote/local are all appropriate for this value.
    When I suddenly get a number of connections (5-10), or when there are
    more than 15-20 concurrent sessions in progress, I begin to get an error
    message from tcpserver and also from qmail-scanner: "Cannot fork:
    Resource temporarily unavailable".

    Using truss, I see that the processes are just sitting there doing
    vfork/nanosleep over and over and over, in a "Resource temporarily
    unavailable" loop. Once the processes get into this state, they don't
    get out of it, and it seems to build up over time (??) - i.e. more and
    more forking errors until everything is hosed. The only way to fix it is
    to kill off all the qmail-scanner processes.

    I have tried messing with every sysctl value I know to change. I've
    checked kern.maxproc, kern.maxfiles, kern.maxfilesperproc,
    kern.maxprocperuid, etc. I've also looked at kern.somaxconn, etc, mostly
    just shooting in the dark. All of these are at sane limits and I am
    nowhere near the limits on open files or the number of processes. I've
    also checked the max datasize (kern.maxdsize), etc etc. I am nowhere
    near the maximum of open files or processes (both of which are
    insanely high: maxusers = 384).

    At the moment I'm not even using softlimit in the qmail startup script
    in order to eliminate it as a cause. tcpserver is set to allow 60
    concurrent qmail processes; the problems start to occur at about 20
    processes, or when there is a very sudden increase in the number of
    connections At the moment there are no set limits on resource
    consumption for tcpserver or its children - at least not ones that I am
    setting!

    Return of limits

    Resource limits (current):
       cputime infinity secs
       filesize infinity kb
       datasize 1572864 kb
       stacksize 262144 kb
       coredumpsize infinity kb
       memoryuse infinity kb
       memorylocked infinity kb
       maxprocesses 5547
       openfiles 11095
       sbsize infinity bytes
       vmemoryuse infinity kb

    Return of limits -U {qscand|qmaild} (no surprise here as I checked
    login.conf)

    Resource limits for class default:
       cputime infinity secs
       filesize infinity kb
       datasize infinity kb
       stacksize infinity kb
       coredumpsize infinity kb
       memoryuse infinity kb
       memorylocked infinity kb
       maxprocesses infinity
       openfiles infinity
       sbsize infinity bytes
       vmemoryuse infinity kb

    Return of ulimit as qmaild/qscand (qmail / qmail-scanner users)

    core file size (blocks, -c) unlimited
    data seg size (kbytes, -d) 1572864
    file size (blocks, -f) unlimited
    max locked memory (kbytes, -l) unlimited
    max memory size (kbytes, -m) unlimited
    open files (-n) 11095
    pipe size (512 bytes, -p) 1
    stack size (kbytes, -s) 262144
    cpu time (seconds, -t) unlimited
    max user processes (-u) 5547
    virtual memory (kbytes, -v) unlimited

    Related sysctls:

    kern.maxproc: 6164
    kern.maxprocperuid: 5547
    kern.maxfiles: 12328
    kern.maxfilesperproc: 11095
    kern.maxusers: 384

    Note, I'm nowhere near these limits:

    kern.openfiles: 421
    ps -aux | wc -l returns 145

    What else could be causing these problems? I would be very grateful for
    any assistance anyone can provide!

    -Justin

    _______________________________________________
    freebsd-questions@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-questions
    To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"

  • Next message: Paul Seniura: "Re: I have some questions about telnet/telnetd/libtelnet/tn3270 and why FreeBSD is different than other BSDs in this regard"

    Relevant Pages

    • limits puzzle - different limits on similar machines
      ... the limits for the user are higher on the working machine ... coredumpsize infinity kb ...
      (freebsd-questions)
    • Re: sysctl kern.ipc.somaxconn limit 65535 why?
      ... limits, which can be adjusted if needed and appropriate. ... over a single queue is almost certain to result in horrible ... bind socket, then spawn a certain amount of children to do the ... client counterpart to create 100k of connections to parents listen ...
      (freebsd-current)
    • Re: file descripter leak in current with Qmail?
      ... When I shut down qmail, ... Several resource limits impact ... a reference counting bug or race). ...
      (freebsd-current)
    • Help in tracking down resource leak
      ... you have some sort of handle leak. ... does it report any resource leaks. ... Does anyone know any good tools for monitoring process quota, ... Are quota limits strictly some sort of memory limits, ...
      (microsoft.public.vc.debugger)
    • Re: a language is a language
      ... Any particular reason that the value has to be less than infinity? ... the lines of "The C standard imposes limits, therefore the C language ...
      (comp.programming)