Horrible PostgreSQL performance with NFS

Hello everyone,

A couple of days ago I've moved our production database from local
disks to NetAPP filer serving NFS. Performance for this server dropped
by factor of 10 if not more. From a happy 10% load, the server hit the
ceiling and sees load of 100% all the time with runqueue above 30. The
server is a Sun V40z machine, 4x Opteron 852, 16GB RAM, tuned for 1600
concurrent connections. I'm running FreeBSD 6.0 BETA5. I don't see any
problem with the network bandwidth or I/O opcount. Total bandwidth
generated by the machine tops at 10Mbps. NICs are bge configured for
100Mbps full-duplex. Filer hardly notices the traffic. It stays around
1% CPU load and 500 NFS ops/s including static file transfers. This
particular model can handle around 6000 I/Os per second. Trussing
database processes shows that most of the syscalls are semops. I tried
to search the archives but couldn't find anything useful for such
setup.

I can provide postgresql config. Sysctl knobs remain untouched not
counting basic NFS tweaking. For mount_nfs I tried to change NFS
buffer sizes -r -w. I also tried the -L option to keep the lock
traffic local. On the postgres side, setting fsync = off in
postgresql.conf. It didn't change a thing.

Any advice besides "give up on NFS" is most welcome. I need NetAPP
over NFS for snapshots and easy FS management.

TIA, /S
--
Sławek Żak / UNIX Systems Administrator 
_______________________________________________freebsd-performance@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Unresponsive NFS server, high load, 100% in kernel
    ... connected through a gigabit network interface dedicated for NFS, ... - 100% of this load is in kernel. ... not much different than during normal work. ... I am aware that the 50% DNLC hit rate is referred as 'very bad, ...
    (comp.unix.solaris)
  • Re: XFS (read-only) support committed to CURRENT
    ... I can quickly reboot nfs server because it is using XFS. ... so they get very high IO load. ... > Anyway, I am not tend to start a flame war between ext3 and XFS, I ...
    (freebsd-current)
  • RE: CD 1 boot failure
    ... If so just burn the boot.iso and boot to it and load off the NFS share. ... You are wrong with your question here on the list for stable Fedora Core release discussions. ...
    (Fedora)
  • Re: Installing 80 workstations with PXE
    ... from DHCP, load the kernel into memory, load the kickstart file and ... Does the first PC you start in the group successfully mount the NFS ...
    (comp.os.linux.networking)
  • Re: Nice processes on Unix
    ... The only reason I was being pretty stubborn about it ... yet with 100+ "load averages". ... | fileserver `hard' then the kernel will block uninterruptably for NFS ... for the load average to skyrocket.] ...
    (comp.lang.lisp)