Re: NFS/Sendmail issue since upgrading to 4.9-STABLE

From: Adam Maloney (adam_at_whee.org)
Date: 11/26/03

  • Next message: Jonathan Gilpin: "RE: 4.9 Stable Crashes on SuperMicro with SMP"
    To: <freebsd-stable@freebsd.org>
    Date: Wed, 26 Nov 2003 16:13:08 -0600
    
    

    Sorry for the noise on the list about this. We think we've identified the
    issue.

    We made the change to make the inbound queues local, and the problem occured
    again almost immediately. That ruled out NFS, and left us with only one
    other realistic option.

    This custom application reads a directory list and for each qf file,
    attempts a LOCK_EX. For the past 2 years this has appeared to work fine
    with Sendmail - if Sendmail is still working on the qf file, we aren't able
    to get an exclusive lock, so we go on to the next file, etc. (Maybe Claus
    can confirm some of this behavior?)

    It appears that Sendmail changed from 8.12.9 (FreeBSD 4.8R) to 8.12.10
    (FreeBSD 4.9-STA). What we believe is happening is that the qf file is
    being created, and our application is getting the exclusive lock before
    Sendmail. This makes Sendmail spit out the "resource temporarily
    unavailable" error, since it can't lock the file, and it leaves our
    application with an empty qf file that we are trying to process.

    It will take several more hours of testing before we're relatively confident
    that this is the issue. Hopefully I'll be posting one last time to confirm
    that everything is fixed.

    ----- Original Message -----
    From: "Adam Maloney" <adam@whee.org>
    To: <freebsd-stable@freebsd.org>
    Sent: Wednesday, November 26, 2003 1:40 PM
    Subject: Re: NFS/Sendmail issue since upgrading to 4.9-STABLE

    > Just an update for anyone paying attention to this issue.
    >
    > We are going to temporarily move the inbound Sendmail queue to be local
    > rather than NFS. We'll run like this through the holiday weekend and see
    if
    > the problems we're having continue or not. If they stop happening, then
    we
    > can be pretty confident that some change in the NFS client code is causing
    > this (the problems started when we upgraded the 1st client, before making
    > any changes to the server).
    >
    > We've also noticed another problem that we believe to be NFS related as
    well
    > with these same machines. It started happening at exactly the same time
    > (though less frequently). This 2nd issue is in a custom application that
    we
    > are running. At this time I don't have a lot of detail, but we are fairly
    > confident that it is also NFS related. Making the inbound Sendmail queues
    > local will also isolate this to be NFS or not.
    >
    > I will report back on Monday with our results.
    >
    > I appreciate everyone's input.
    >
    > ----- Original Message -----
    > From: "Adam Maloney" <adam@whee.org>
    > To: <freebsd-stable@freebsd.org>
    > Sent: Wednesday, November 26, 2003 8:16 AM
    > Subject: NFS/Sendmail issue since upgrading to 4.9-STABLE
    >
    >
    > > Hi all,
    > >
    > > I have 3 servers accepting mail, with sendmail in Queue-only mode
    dropping
    > > mail into an NFS queue (mounted from a 4th machine.) All 4 machines
    > > involved are running FreeBSD.
    > >
    > > I upgraded the 3 NFS clients from 4.8 to 4.9-STABLE on Monday, and
    started
    > > seeing this sendmail message:
    > >
    > > Nov 26 06:51:29 unix20 sm-mta[48069]: (real-message-id): SYSERR(root):
    > > queueup: cannot create queue file ./qf(real-message-id), euid=0:
    Resource
    > > temporarily unavailable
    > >
    > > The df(message-id) file is created and appears to be complete, however
    the
    > > qf file is 0 bytes.
    > >
    > > This happens sporadically, a few times a day on each machine (together
    > doing
    > > about 150,000 messages/day).
    > >
    > > I upgraded the NFS server from 4.8 to 4.9 yesterday, thinking that might
    > > help, but the problem is still there.
    > >
    > > Any ideas how to troubleshoot this one?
    > >
    > > _______________________________________________
    > > freebsd-stable@freebsd.org mailing list
    > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
    > > To unsubscribe, send any mail to
    "freebsd-stable-unsubscribe@freebsd.org"
    > >
    >
    > _______________________________________________
    > freebsd-stable@freebsd.org mailing list
    > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
    > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
    >

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  • Next message: Jonathan Gilpin: "RE: 4.9 Stable Crashes on SuperMicro with SMP"

    Relevant Pages

    • Re: NFS/Sendmail issue since upgrading to 4.9-STABLE
      ... We are going to temporarily move the inbound Sendmail queue to be local ... rather than NFS. ... > involved are running FreeBSD. ... > seeing this sendmail message: ...
      (freebsd-stable)
    • Re: sendmail starts before rpc.statd and rpc.lockd
      ... > It looks like sendmail starts before rpc.lockd and rpc.statd? ... This is a nfs server and diskless ... It shouldn't cause diskless clients to hang, or at least, doesn't ... Ctrl-T while a process is attempting to grab an NFS-backed file lock will ...
      (freebsd-current)
    • Re: sendmail starts before rpc.statd and rpc.lockd
      ... > It looks like sendmail starts before rpc.lockd and rpc.statd? ... This is a nfs server and diskless ... It shouldn't cause diskless clients to hang, or at least, doesn't ... Ctrl-T while a process is attempting to grab an NFS-backed file lock will ...
      (freebsd-net)