Re: How to diagnose performance problems in web applications?

From: Gary Mills (mills_at_cc.umanitoba.ca)
Date: 02/22/05 

Date: 22 Feb 2005 21:25:53 GMT

In <vilain-A32073.11314922022005@comcast.dca.giganews.com> Michael Vilain <vilain@spamcop.net> writes:

>In article <cvfn8p$9o1$1@canopus.cc.umanitoba.ca>,
> mills@cc.umanitoba.ca (Gary Mills) wrote:

>> I have a couple of instances of performance problems with web
>> applications. I'm looking for general suggestions on how to
>> determine the source of the problems. Both are with apache web
>> web servers running on SPARC hardware under Solaris 9.
>>
>> The first is an e-mail application using mod_php. People complain
>> that pages are slow to load and refresh. I'd like to determine
>> where most of the delay for HTTP transactions is taking place.
>>
>> The second uses perl CGI scripts that do Sybase transactions.
>> During intervals of high activity, the load average on the machine
>> becomes excessive. In this case, I'd like to determine what is
>> causing the high load average. I'm assuming that there is a
>> bottle neck of some sort. These intervals of high demand only
>> occur a few times a year, and last for a day or two.

>Do you have any experience in system performance tuning and capacity
>planning? How about database tuning?

Some, and our DBMs do as well.

>Do you have any sort of performance monitoring software running on the
>system to collect utilization statistics for trend analysis, like for
>example, sar. Without information on what your system is doing when
>people complain "it's slow", you really can't do anything useful other
>than say "try again later".

In the case of the e-mail application, I don't think that statistics
will help. Response to HTTP transactions is always slow, even when
the machine is idle. I'd just like to determine what portion of the
server-side processing is occupying most of the time.

In the case of the Sybase transactions, I agree that collecting
statistics at the time of the high demand would be a useful approach.
I'm going to suggest that. I don't know if much can be done in
between these occasions.

>You might start with the basic. Adrian Cockroft's book, although dated
>and not directly relevant to web applications, still will give you
>basics of how Solaris does what it does. You can apply this, along with
>performance monitoring information from tools, to determine where the
>bottleneck is on your system.

Yes, I've read that book. What I'm really looking for is process
instrumentation. Last time we had a similar problem, I just used
`truss' with timestamps, and discovered that the e-mail application
was attempting to authenticate to the IMAP server with an incorrect
administrator password on every transaction. The server imposes a
time delay after authentication failures. This caused very slow
response to transactions.

>Some links:

>http://www.sun.com/980922/tuning2/about.html
>http://www.sun.com/sun-on-net/itworld/UIR010329cockcroftletters.html
>http://www.sun.com/sun-on-net/performance/se3/
>http://www.sun.com/951001/columns/adrian/column2.html

Thanks.

>One thing to keep in mind, many complaints about performance are
>frequently due to the application either not being designed to scale or
>being poorly designed from the start. Until you have data that points
>to Solaris being the problem, I'd focus on what you're doing in the php
>code and the CGI/Sybase code. Chances are, it's there.

The people who designed the application want to solve the problem
by throwing away Solaris and the SPARC hardware, replacing it with
a Linux cluster on Intel hardware. I'd prefer to identify the real
problem, which may indeed turn out to be poor design. That's why
I'm looking for diagnostic techniques. 

-- 
-Gary Mills-    -Unix Support-    -U of M Academic Computing and Networking-