Re: amrd disk performance drop after running under high load

Alexey Popov wrote:
Hi

Kris Kennaway wrote:
So I can conclude that FreeBSD has a long standing bug in VM that could be triggered when serving large amount of static data (much bigger than memory size) on high rates. Possibly this only applies to large files like mp3 or video.
It is possible, we have further work to do to conclude this though.
I forgot to mention I have pmc and kgmon profiling for good and bad times. But I have not enough knowledge to interpret it right and not sure if it can help.
pmc would be useful.
pmc profiling attached.

Sorry for the delay, I was travelling last weekend and it took a few days to catch up.

OK, the pmc traces do seem to show that it's not a lock contention issue. That being the case I don't think the fact that different servers perform better is directly related. In my tests multithreaded web servers don't seem to perform well anyway.

There is also no evidence of a VM problem. What your vmstat and pmc traces show is that your system really isn't doing much work at all, relatively speaking.

There is also still no evidence of a disk problem. In fact your disk seems to be almost idle in both cases you provided, only doing between 1 and 10 operations per second, which is trivial.

In the "good" case you are getting a much higher interrupt rate but with the data you provided I can't tell where from. You need to run vmstat -i at regular intervals (e.g. every 10 seconds for a minute) during the "good" and "bad" times, since it only provides counters and an average rate over the uptime of the system.

What there is evidence of is an interrupt aliasing problem between em and USB:

irq16: uhci0 1464547796 1870
irq64: em0 1463513610 1869

This is a problem on some intel systems. Basically each em0 interrupt is also causing a bogus interrupt to the uhci0 device too. This will be causing some overhead and might be contributing to the UMA problems. I am not sure if it is the main issue, although it could be. It is mostly serious when both irqs run under Giant, because they will both fight for it every time one of them interrupts. That is not the case here but it could be other bad scenarios too. You could try disabling USB support in your kernel since you dont seem to be using it.

Kris
_______________________________________________
freebsd-stable@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: amrd disk performance drop after running under high load
    ... could be triggered when serving large amount of static data on high rates. ... I forgot to mention I have pmc and kgmon profiling for good and bad times. ... There is also still no evidence of a disk problem. ... In the "good" case you are getting a much higher interrupt rate but with the data you provided I can't tell where from. ...
    (freebsd-hackers)
  • Re: amrd disk performance drop after running under high load
    ... could be triggered when serving large amount of static data on high rates. ... I forgot to mention I have pmc and kgmon profiling for good and bad times. ... I believe the reason of high interrupt rate in "good" cases is that server sends much traffic. ...
    (freebsd-hackers)
  • Re: amrd disk performance drop after running under high load
    ... could be triggered when serving large amount of static data on high rates. ... I forgot to mention I have pmc and kgmon profiling for good and bad times. ... I believe the reason of high interrupt rate in "good" cases is that server sends much traffic. ...
    (freebsd-stable)
  • Re: amrd disk performance drop after running under high load
    ... could be triggered when serving large amount of static data on high rates. ... I forgot to mention I have pmc and kgmon profiling for good and bad times. ... the pmc traces do seem to show that it's not a lock contention issue. ...
    (freebsd-hackers)
  • Re: amrd disk performance drop after running under high load
    ... could be triggered when serving large amount of static data on high rates. ... I forgot to mention I have pmc and kgmon profiling for good and bad times. ... the pmc traces do seem to show that it's not a lock contention issue. ...
    (freebsd-stable)