nfs server overload (nfsd)

Hello,

We are expecting incredible overload in a NFS server. A top shows nfsd
consuming most of the CPU:

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND
6000 root -8 0 1204K 660K biord 1 124:15 27.88% 27.88% nfsd
6002 root 4 0 1204K 660K *Giant 0 124:18 17.58% 17.58% nfsd
6006 root 4 0 1204K 660K *Giant 0 123:38 10.21% 10.21% nfsd
6005 root 4 0 1204K 660K *Giant 0 123:36 7.47% 7.47% nfsd
6003 root 4 0 1204K 660K *Giant 0 123:08 4.15% 4.15% nfsd
6001 root 4 0 1204K 660K *Giant 0 123:16 2.83% 2.83% nfsd

Memory looks fine:

Mem: 27M Active, 910M Inact, 136M Wired, 51M Cache, 112M Buf, 1828K Free
Swap: 2048M Total, 72K Used, 2048M Free

Typing in the nfs server (console/ssh) becomes terrible, the server does
not reply well.

We are running this nfs server in FreeBSD 5.3-RELEASE-p23 on a Compaq
Proliant server with a Compaq Smart Array 5300 that comunicates with a
array of disks:

/dev/da0s1d 164G 124G 27G 82% /data0
/dev/da1s1d 131G 80G 41G 66% /data1

We have /data0 and /data1 exported:

/data0 -maproot=root -alldirs -network 192.168.62.0 -mask 255.255.255.0
/data1 -maproot=root -alldirs -network 192.168.62.0 -mask 255.255.255.0

so a couple of incoming SMTP servers we have can deliver e-mail to
those filesystems.

We are running exim 4.60.0 in those other servers, 4.10-RELEASE-p5 in
one of them, and FreeBSD 6.0-RELEASE #0 in the other one.

If we stop exim delivering e-mail, nfs server does well, the cpu gets
free, and the nfs server works fine (replies to user interaction, etc).

FreeBSD 6.0 sysctl output (nfs related):

vfs.nfs4.access_cache_timeout: 60
vfs.nfs4.nfsv3_commit_on_close: 0
vfs.nfs.downdelayinitial: 12
vfs.nfs.downdelayinterval: 30
vfs.nfs.realign_test: 1294030
vfs.nfs.realign_count: 0
vfs.nfs.bufpackets: 4
vfs.nfs.reconnects: 2
vfs.nfs.iodmaxidle: 120
vfs.nfs.iodmin: 4
vfs.nfs.iodmax: 20
vfs.nfs.defect: 0
vfs.nfs.nfs_ip_paranoia: 1
vfs.nfs.diskless_valid: 0
vfs.nfs.diskless_rootpath:
vfs.nfs.access_cache_timeout: 2
vfs.nfs.nfsv3_commit_on_close: 0
vfs.nfs.clean_pages_on_close: 1
vfs.nfs.nfs_directio_enable: 0
vfs.nfs.nfs_directio_allow_mmap: 1
vfs.nfsrv.nfs_privport: 0
vfs.nfsrv.async: 0
vfs.nfsrv.commit_blks: 0
vfs.nfsrv.commit_miss: 0
vfs.nfsrv.realign_test: 0
vfs.nfsrv.realign_count: 0
vfs.nfsrv.gatherdelay: 10000
vfs.nfsrv.gatherdelay_v3: 0

FreeBSD 4.10 sysctl output (nfs related):

vfs.nfs.nfs_privport: 0
vfs.nfs.async: 0
vfs.nfs.commit_blks: 0
vfs.nfs.commit_miss: 0
vfs.nfs.realign_test: 84602323
vfs.nfs.realign_count: 99713
vfs.nfs.bufpackets: 4
vfs.nfs.gatherdelay: 10000
vfs.nfs.gatherdelay_v3: 0
vfs.nfs.defect: 0
vfs.nfs.nfs_ip_paranoia: 1
vfs.nfs.diskless_valid: 0
vfs.nfs.diskless_rootpath:
vfs.nfs.diskless_swappath:
vfs.nfs.access_cache_timeout: 2
vfs.nfs.nfsv3_commit_on_close: 0

This couple of servers mounts the filesystems with this options:

192.168.62.54:/data1 /mail nfs
rw,nfsv3,intr,dumbtimer,rdirplus,nosuid,nodev 0 0
192.168.62.54:/data0 /data0 nfs
rw,nfsv3,intr,dumbtimer,rdirplus,nosuid,nodev 0 0

On the server, sysctl nfs related output looks like this:

vfs.nfs.downdelayinitial: 12
vfs.nfs.downdelayinterval: 30
vfs.nfs.realign_test: 2694
vfs.nfs.realign_count: 0
vfs.nfs.bufpackets: 4
vfs.nfs.reconnects: 2
vfs.nfs.iodmaxidle: 120
vfs.nfs.iodmin: 4

vfs.nfs.iodmax: 20
vfs.nfs.defect: 0
vfs.nfs.nfs_ip_paranoia: 1
vfs.nfs.diskless_valid: 0
vfs.nfs.diskless_rootpath:
vfs.nfs.access_cache_timeout: 2
vfs.nfs.nfsv3_commit_on_close: 0
vfs.nfs4.access_cache_timeout: 60
vfs.nfs4.nfsv3_commit_on_close: 0
vfs.nfsrv.nfs_privport: 0
vfs.nfsrv.async: 1
vfs.nfsrv.commit_blks: 579238
vfs.nfsrv.commit_miss: 413059
vfs.nfsrv.realign_test: 88269083
vfs.nfsrv.realign_count: 11961
vfs.nfsrv.gatherdelay: 10000
vfs.nfsrv.gatherdelay_v3: 0
debug.hashstat.nfsnode: 65536 5 1 0

Thanks in advance,

Best regards,
Angel Blazquez
_______________________________________________
freebsd-performance@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: NFS Locking Issue
    ... to FreeBSD 6.x and later. ... Turn off rpc.lockd on either the server or client before the cp command, ... At one point we had in our test network a 6.1 NFS server providing files to 5.4 diskless clients without any problems. ...
    (freebsd-stable)
  • Re: Fw: get me off this list
    ... headers and see what server is passing the message on to gmail. ... Verifying that I have SMP up and running ... FreeBSD is a registered trademark of The FreeBSD Foundation. ... CPU: Dual Core AMD OpteronProcessor 180 (2412.38-MHz 686-class ...
    (freebsd-questions)
  • RELENG 6.1/AMD64 system hangs when SMP enabled
    ... I'm rather new to freebsd so forgive me if I am saying things that don't make sense. ... When I put heavy load on the server the server sometimes hangs. ... CPU: AMD OpteronProcessor 246 ... real memory = 2147418112 ...
    (freebsd-stable)
  • Re: 100Mbit network performance - again
    ... > using realtek, and freebsd can get the same but uses less cpu in doing ... What server are you serving HTTP traffic with and what are ... But the real question is what sort of CPU speed/RAM/HD speeds and what ...
    (freebsd-questions)
  • Fwd: FreeBSD 6.2-STABLE && apache 2.2.4 = bad performance. Help!
    ... Please don't cross-post between multiple FreeBSD lists; ... > Right after starting apache, the loads on the server will climb to ... The fact that your server starts dogging out around 40 processes is ... > cpu server and it's performing a lot worse than some servers I have ...
    (freebsd-questions)