Re: How do you respond to "It's slow"?
- From: stoltzfu@xxxxxxxxxxxxxxxxxxxxx (Mark Stoltzfus)
- Date: Thu, 16 Mar 2006 20:28:32 +0000 (UTC)
In article <1142435902.206052.140820@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>,
<tonij67@xxxxxxxxxxx> wrote:
Actually, I DO have tons of trending data. I use a combination of SAR,
Orca and custom performance scripts (iostat, netstat, vmstat, etc). In
just about every instance I never see anything out of the ordinary when
these claims are made. It becomes more of a political issue; the
complainers are screaming louder than me, and it doesnt help that the
usual suspects are basically bitter and look for things to bitch about
:( Several times I have provided multi page reports, complete with
pretty graphs showing system health levels. The management is about as
non techincal as you can get, so they look at these things and go
"whatever" while the screamers are coming back with brilliant rebuttles
like "I don't agree with those charts". Then it comes down to a me vs.
them situation, which I really hate. It's an uphill battle all the
way.
For the past several years, I'd been working for a rather large company
that was/is having significant performance problems with its homegrown
J2EE CRM tool, running on Iplanet/Weblogic/DB2 on the web/app/db tiers,
respectively. The application itself was/is a complete mess, and it made
performance troubleshooting an adventure, to say the least. I feel your
frustration. Nevertheless, this was an invaluable learning opportunity to
really figure out how the different tiers interact. Here are some of
the lessons I learned:
1) First and foremost, walk/drive/fly to where the users are, sit down
with them, and find out how they use the application. Get as best an
idea as possible as to how they use the application, and talk to as many
people as possible about how the slowness manifests itself.
2) Build a test environment, and have the users beat up on it. Most
people can't afford identical hardware, but try at least to have identical
software. Several times we were bitten by performance problems
associated with specific releases of software that differed between our
test and production systems.
3) Be as familiar as possible with all the different tiers involved. At
one point, we had users saying "the application freezes for 30 seconds
at a time". We looked at all of our performance data, and nothing stood
out, so we assumed they were exagerrating. That is, until we realized
that a full java garbage collection on a 2GB heap on an E4500 takes about 30
seconds, in which the entire application stops. This ties into my next
lesson.
4) Wherever possible, collect network, system, application AND database
performance data. You already run Orca, so you've got the system data
covered. We ran Orca on a 60-second interval and it proved invaluable
over and over again. Several vendors (Veritas, IBM, Oracle) offer
excellent solutions to give you fantastic insight into what queries are
hogging up which resources in the DB and on the system. We made huge
gains by indexing trouble tables, writing stored procedures, and rewriting
bad queries. Sit down with the people responsible for each tier and
look at every layer. In the course of our testing, we discovered
issues at every single tier, including:
a) Screens on the user's desktops were too small, they were actually
slowed down because they had to scroll around just to see the entire
page.
b) Network bandwidth between remote offices and corporate couldn't
support someone surfing for their next dream home and someone else
actually trying to use the application (all web traffic went through
a central proxy server).
c) The database drivers on the application tier had a known bug where
they got slower over time.
d) The database wasn't up-to-date on patches (more specifically,
n 2005, it was still running Solaris 8, kernel rev 108528-15. We
got a 30% improvement in transactions times simply by installing
the latest 8_Recommended).
e) The database itself wasn't tuned properly (too many tweaks here
to note).
It is an end-to-end process. I added point "a" above to illustrate that
sometimes the bottlenecks are non-technical issues. That being said, point
#1 turned out to be the most important, because that's where it all began.
It really sucks.
It does, sometimes, but when you correct persistent performance problems,
you'll never see your users (or management) more grateful. Productivity
translates directly into dollars.
Hope that helps,
Mark
.
- References:
- How do you respond to "It's slow"?
- From: tonij67
- Re: How do you respond to "It's slow"?
- From: JohnK
- Re: How do you respond to "It's slow"?
- From: tonij67
- How do you respond to "It's slow"?
- Prev by Date: Re: Enabling large text pages on Solaris 10/11
- Next by Date: Re: ipfilter why does moving a rule break it?
- Previous by thread: Re: How do you respond to "It's slow"?
- Next by thread: Re: How do you respond to "It's slow"?
- Index(es):
Relevant Pages
|
