Sockets stuck in FIN_WAIT_1

I have a rather busy Apache 2.2 server; tons of small & some large requests. It's a standard Dell 2650 server using the bge (broadcom) network driver.

I seem to have a rather strange problem where after just a day or so Apache just stops processing new connections. You can connect to port 80, but trying to get Apache to process any data just hangs. There is nothing strange in dmesg or in /var/log/messages.

The server has plenty free available physical RAM, swap is untouched, CPU load is low, etc. Apache is setup to handle a max of 100 clients using prefork model.

If I stop and restart Apache, it does not help.

What I do notice is 1000's of sockets stuck in "FIN_WAIT_1" in netstat:

[web0:~] netstat -an | grep FIN_WAIT | wc -l
1827

These stick around forever. Some eventually trickle away after hours, but the only thing that appears to fix it is to reboot the server. Then all is fine for another day or so. I've tried just about every tuning trick out there but to no eval. I can mitigate the problem by increasing available socket buffs and decreasing the tcp.sendspace. I've tried different versions of Apache and I've tried with and without the accf_http kernel filter.

Here is what I have on the server now:

sysctl.conf:

kern.maxfiles=65535
kern.maxfilesperproc=16384
kern.ipc.maxsockbuf=4194304
kern.ipc.somaxconn=1024
net.inet.tcp.sendspace=8192
net.inet.tcp.recvspace=8192
net.inet.tcp.keepidle=900000
net.inet.tcp.keepintvl=30000
net.inet.tcp.msl=5000
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.tcp.inflight_enable=1


and loader.conf

accf_http_load="YES"
kern.ipc.nmbclusters=32768
net.inet.tcp.tcbhashsize=4096
kern.ipc.maxsockets=131072


ipfw:

00200 allow tcp from any to me 80 setup
00200 allow icmp from any to me icmptype 0,3,8,11
00200 deny log ip from any to me


ifconfig:
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=3<RXCSUM,TXCSUM>
inet 1.2.3.4 netmask 0xfffffff8 broadcast 5.6.7.8
ether 00:06:5b:f7:c8:7b
media: Ethernet autoselect (1000baseTX <full-duplex>)



Any ideas would be greatly appreciated.

--
Robert Blayzor, BOFH
INOC, LLC
rblayzor@xxxxxxxx
http://www.inoc.net/~rblayzor/



_______________________________________________
freebsd-stable@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • RPC over HTTPS reversed proxy configuration...
    ... I am trying to setup RPC over HTTPs via an apache reverse proxy. ... OWA and OMA without issue to the exchange server. ...
    (microsoft.public.exchange.connectivity)
  • Re: Apache startup resolve problem
    ... > complete bind configuration file and the VirtualHost snippet of the ... > Apache server you have a problem with. ... The bind setup on 192.168.1.3: ...
    (comp.infosystems.www.servers.unix)
  • Re: apache question
    ... # Based upon the NCSA server configuration files originally by Rob McCool. ... # configuration directives that give the server its instructions. ... Directives that control the operation of the Apache server process as ...
    (alt.php)
  • Re: Apache and SSL
    ... # Based upon the NCSA server configuration files originally by Rob McCool. ... # This is the main Apache server configuration file. ... # configuration directives that give the server its instructions. ...
    (RedHat)
  • Re: Apache vs IIS
    ... Windows Server not on my Linux Server so there for I would chose IIS. ... Not that Apache is bad but ASP.NET is far easier and faster to create good web forms in. ... PHP on a IIS server is rather easy to run once you install PHP on a PC but if you only use PHP why not use Apache for Windows. ...
    (alt.php)