Re: amrd disk performance drop after running under high load

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexey Popov wrote:
Hi.

Kris Kennaway wrote:

After some time of running under high load disk performance become
expremely poor. At that periods 'systat -vm 1' shows something like
this:
What does "high load" mean? You need to explain the system workload
more.
This web service is similiar to YouTube. This server is video store. I
have around 200G of *.flv (flash video) files on the server.

I run lighttpd as a web server. Disk load is usually around 50%, network
output 100Mbit/s, 100 simultaneous connections. CPU is mostly idle.

As you can see it is a trivial service - sending files to network via HTTP.


Disks amrd0
KB/t 85.39
tps 5
MB/s 0.38
% busy 99

Apart of all, I tried to make mutex profiling and here's the results
(sorted by the total number of acquisitions):

Bad case:

102 223514 273977 0 14689 1651568 /usr/src/sys/vm/uma_core.c:2349 (512)
950 263099 273968 0 15004 14427 /usr/src/sys/vm/uma_core.c:2450 (512)
108 150422 175840 0 10978 22988519 /usr/src/sys/vm/uma_core.c:1888
(mbuf)

> Here you can see that high UMA activity happens in periods of low disk
> performance. But I'm not sure whether this is a root of the
problem, not
> a consequence.

The extremely high contention there does seem to say you have a mbuf
starvation problem and not a disk problem. I don't know why this
would be happening off-hand.
But there's no mbuf shortage in `netstat -m`.

What else can I try to track down the source of the problem?

Can you also provide more details about the system hardware and
configuration?
This is Dell 2850 2 x Xeon 3.2, 4Gb RAM, 6x300Gb SCSI RAID5. I'll attach
details.

With best regards,
Alexey Popov


------------------------------------------------------------------------

_______________________________________________
freebsd-stable@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@xxxxxxxxxxx"


last pid: 11008; load averages: 0.07, 0.10, 0.08 up 47+08:32:50
11:46:15
38 processes: 1 running, 37 sleeping

Mem: 46M Active, 3443M Inact, 246M Wired, 144M Cache, 208M Buf, 5596K Free
Swap: 2048M Total, 4K Used, 2048M Free


PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
56386 root 1 4 0 19856K 10000K kqread 1 115:19 2.88% lighttpd
636 root 1 96 0 18292K 4212K select 0 25:39 0.00% snmpd
784 root 1 96 0 19668K 2072K select 1 2:31 0.00% sshd
680 root 1 96 0 7732K 1384K select 0 1:59 0.00% ntpd
1540 root 1 96 0 35092K 6496K select 0 1:30 0.00% httpd
769 root 4 20 0 14148K 2632K kserel 0 1:04 0.00% bacula-fd
755 root 1 96 0 3852K 1060K select 1 0:22 0.00% master
568 root 1 96 0 3648K 908K select 0 0:18 0.00% syslogd
80663 root 1 8 0 3688K 1016K nanslp 1 0:05 0.00% cron
760 postfix 1 96 0 3944K 1160K select 0 0:04 0.00% qmgr
89776 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd
89763 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd
89774 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd
89775 www 1 96 0 35180K 6684K select 0 0:04 0.00% httpd
699 root 1 20 0 7732K 1388K pause 0 0:03 0.00% ntpd
484 root 1 4 0 652K 220K select 0 0:00 0.00% devd
10904 llp 1 96 0 30616K 3564K select 0 0:00 0.00% sshd
10915 root 1 20 0 3912K 2340K pause 1 0:00 0.00% csh


You run apache with mod_perl or php too. How many clients handle this
apache server? Also in this light load you have locked files! Check
script execution times (/server-status may be useful).
When you have hight load check swap usage and haw many processes are in
lockf state.

Best Regards


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFHFHuVxJBWvpalMpkRAnqTAJ9FgURNk98dtD0HYX6xIz17R6sLpQCgh5nJ
XBtfOyzJJbkjzVzSF/WfmHc=
=oTHZ
-----END PGP SIGNATURE-----
_______________________________________________
freebsd-hackers@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • RE: Web Service Implementation Security Question
    ... Server was unable to process ... CompilerParameters parameters, Assembly assembly, Hashtable assemblies) at ... > As for the security problems regarding on using TypedDAtaset in asp.net ... Web Service Implementation Security Question ...
    (microsoft.public.inetserver.iis.security)
  • Re: amrd disk performance drop after running under high load
    ... This web service is similiar to YouTube. ... This server is video store. ... Also in this light load you have locked files! ...
    (freebsd-stable)
  • Re: Permission Errors
    ... >a domain account so that it can be validated on the win2k server machine. ... However I have discovered that I can deploy the web service to the W2K ... domain server and there it runs with no problems, ... for me to decide that the new XP dev machine is up and usable. ...
    (microsoft.public.dotnet.general)
  • Re: Permission Errors
    ... I built the TempConvert3 sample web service on the XP work ... Next I looked at the 2003 server and saw that the local path was ... XP work station. ... >>local workstation, not the server. ...
    (microsoft.public.dotnet.general)
  • Re: Permission Errors
    ... So when we move a webproject to a new server (creating the new virtual ... I built the TempConvert3 sample web service on the XP work ... |>>local workstation, not the server. ... visual studio automatically selected my work station as the ...
    (microsoft.public.dotnet.general)