Re: Need performance help
From: Karl Rohwedder (emil.mustermann_at_t-online.de)
Date: 02/21/05
- Next message: JF Mezei: "Module CREDEF inconsistency between DEC-C and Macro"
- Previous message: Paul Sture: "Re: Micro$oft warns of undetectable spyware security risk ..."
- In reply to: David J Dachtera: "Re: Need performance help"
- Next in thread: David J Dachtera: "Re: Need performance help"
- Reply: David J Dachtera: "Re: Need performance help"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Mon, 21 Feb 2005 09:20:07 +0100
David J Dachtera wrote:
> "Main, Kerry" wrote:
>
>>>-----Original Message-----
>>>From: David J Dachtera [mailto:djesys.nospam@comcast.net]
>>>Sent: February 20, 2005 11:48 AM
>>>To: Info-VAX@Mvb.Saic.Com
>>>Subject: Re: Need performance help
>>>
>>
>>[snip...]
>>
>>
>>>>Something to consider is to set up web site on Apache CSWS which has
>>>>daily / weekly reports that the end users require. That way, you can
>>>>simply tell them to check the web site for the latest reports.
>>>>
>>>>As I recall, there has been a few suggestions in the past
>>>
>>>on how to set
>>>
>>>>this up.
>>>
>>>...and by what magic do these reports get created so the users can
>>>display them? Seems to me key software layers remain missing.
>>>
>>
>>Reports and graphs can be generated automatically by any number of free
>>or commercial pkgs. Batch job then simply copies these reports/graphs to
>>web site (in appropriate format) where they are available for web
>>viewing by BU's and end users.
>>
>>As an example ECP (free) provides the capability to generate reports and
>>graphs.
>>http://h71000.www7.hp.com/openvms/products/ecp/index.html
>
>
> ...which brings us back to my earlier comment:
>
> "Yeah, looked at that too. Don't remember all the specifics, but I seem
> to recall the automatable portions only offering the equivalent of the
> first page (tabular summary) of an SPM report (no histograms?)."
>
> If you, as an OpenVMS ambassador, can provide any deeper insight, such
> as how to get SPM/PSPA-like ACSII histograms out of ECP, or how to
> automate TLViz, the result might be a disfavor to CA, but a very *LARGE*
> favor to the OpenVMS community at large.
>
I wrote a little piece of DCL, which uses the CSVPNG image to create a nice
HTML report of yesterdays T4 data.
This is then presented to the our users via a simple HTML page.
Code snippet:
$ Spawn/Wait/Input=NL:/Output='Dev''Direc'T4_CSVPNG_'Node'_'Yesterday'.LOG -
Csvpng 'Tmp' -w=800 -h=400 -l -b=1 -a=30 -t=06:00,20:00 -i -
-p=[mon.mode],split,[mon.syst],XFC,[mon.disk],[mon.page] -
-u='Yesterday'_'Node'
The CSVPNG ist not part of T4, I received it after mailing Steve Lieman after
attending the Technical Updates in Germany. I'vw attached the Readme.
mfg Kalle
CSVPNG V1.0-65 Readme 5-Aug-2004
This program generates line graphs from OpenVMS T4 CSV Files. T4 is a tool
developed by OpenVMS Engineering which monitors a system and converts the
Monitor utility output to Comma Separated Value format files.
CSVPNG is available for Alpha and IPF versions of OpenVMS, as well as for
Windows 2000/XP.
To use the OpenVMS version you must first define a 'foreign command' as follows:
$CSVPNG :== $<disk:><dir>CSVPNG_ALPHA.EXE
If processing very large CSV files on an OpenVMS system you may need to increase
PGFLQUOTA to about 800000.
The general format of a CSVPNG command is:
csvpng [options] <csv file 1> [csv file2] ...
At least one CSV file must be specified. If more than one file is specified,
the matching parameters from all the files are shown on the same chart.
Options may be specified using either '-' or '/' as a prefix. If an option takes
a value, use '=' to specify. Each option must be preceded by a space.
The generated charts are .PNG files and a HTML file, CSVPNG.HTM, is generated
which will display all the graphs. On a Windows system, view CSVPNG.HTM in a
browser to see the charts. On OpenVMS, view the html file using a suitable
browser, e.g. Mozilla or via a web browser from a Windows system
Options:
/average[=val][:0]
Moving Average. This specifies the moving average window size (in
minutes) to use. The moving average line will be shown on the chart in
red. The peak period is shown (shaded) in the CSVPNG output. Typical
values to use for moving average are 10 or 60 mins. Append ':0' to
suppress the full line graph and show only the Moving Average graph.
If /a is used without specifying a value, the average is plotted.
/a cannot be used with overlay or multifile charts.
If the /extract qualifier is also used, a new CSV file is generated with
the moving average values.
/border[=val]
Border. If /b is used, the charts will have a gray border. The optional
values specifies the chart background: 0 black, 1 white.
/correlate=val[:th]
Correlate. The parameter corresponding to the value supplied is
checked for correlation with all the parameters specified by /p.
Optionally, the correlation threshold can be specified by adding
:<threshold>. The default correlation threshold is .65.
Example: /c=3:.75 would check parameter 3 from the CSV file against
the parameters specified by /p using a threshold of .75. A list of the
parameters meeting the correlation criteria is written to CORREL.LIS.
To check correlation of all parameters use /c=0 or /c=ALL. For example,
to check for exact correlation of all parameters use /c=ALL:1 /p=ALL.
/dots[=size][:color]
Dots. Prints a dot at each data point. The size of the dot can be
set by using /d=<val>, The default is a small dot (val=1). The color of
the dot can be specified by using the syntax /d=<val>:<color> where
<color> is one of red, green, blue, magenta, yellow (first character
is sufficient), or use k for black.
Example: /d=2:red gives a round Red dot of 2 pixels radius
/d=3:k gives a slightly bigger Black dot
/extract[=file][:n]
Extract. The parameters specified by /p, and matching the range
specified by /t, are written to the specified file. If no file
specification is given, T4_OUT.CSV is used.
If a value 'n' is specified, only every 'n'th record will be output.
If /a=n is also used, the average of 'n' records will be written to the
new CSV file. Note this average feature does not work with /x because no
calculations are done when /x is used.
See also /x
/fill
Fills in the area under each graph with a solid color.
/height=height
Specifies the required chart height in pixels.
/i
Will not update or create History file. Also the PARAM.LIS, T4_LOG.LIS
and T4_PEAK files are not created.
/j[=val]
Exclude charts with constant data less than 'val'. If no value is
specified 1 is used. If /j is not used, all charts with constant
data are excluded. Use this qualifier if you want to chart some
parameters which are constant throughout the sampling period.
/k="text":xy
Caption. Use this option to specify a line of text to be written at
the top of the chart. A maximum of 80 characters can be used.
The font size and color can be controlled using this convention:
Append :xy to the text string where 'x' is a code for Font size and
y is a code for Font color. Available codes are:
Font size: (T)iny, (S)mall, (L)arge [default], (M)ediumBold, (G)iant
Font color: (R)ed [default], (B)lack, (G)reen, (Y)ellow, bl(U)e
For example:
/k="This is a sample Caption using Large Red Font":LR
/k="This is a sample Caption using tiny blue Font":TU
/lines[=value][:thickness][:color]
Prints grid lines on the chart and show a vertical scale. If a
value (in minutes) is not specified, an appropriate default is used.
The chart line thickness and color can specified using this syntax:
/l=<value>:<thickness>:<color> where thickness determines the chart line
thickness (0 is the default value) and color is one of red, green, blue,
magenta, yellow (first character is sufficient), or use k for black.
Examples:
/l=60 gives vertical lines every 60 minutes, default thickness and color
/l=15:1:g gives a thicker green chart line with vertical lines every 15 mins
/l=10:0:r changes the chart line color to Green. The thickness id the default
value and vertical lines are every 10 minutes
/merge[=file]
Combine parameters from two or more CSV files. The output CSV
file will contain parameters from all the specified CSV files with the
values for matching parameters added.
/normalize
For Overlay charts, displays each graph normalized.
/overlay[=val]
The selected parameters will be printed on one chart. A maximum of
15 parameters may be used. If no value is used, the chart legend
will be at the side. If val=1, the legend will be placed at the
bottom of the chart. Do not use this option with multifile charts,
they are automatically overlaid.
/parameters=params
CSV Parameters. If 0 or 'ALL', all parameters from the file(s) will
be graphed. If /p is not specified at all, you will be shown all the
parameter headings in the file and asked to choose. A list of
parameter numbers may be entered, e.g. /p=3,4,5,6. A numeric parameter
range may also be entered, e.g. /p=100-300. The parameters may also be
entered as strings. e.g. if /p=disk is entered, all parameters with
'disk' in the name will be graphed. A number of strings may be entered,
e.g. /p="cpu 00",disk. If a string is preceded by '-' all parameters
containing that string will be excluded. Use quotes with multi-word
strings. The list of parameters used is written to the file PARAM.LIS.
If /p=PARAM.LIS is used, the parameters from that file will be used.
/quiet
Quiet mode. The charts are generated but very little output is shown on
the screen.
/rules=[file]
Specify a Rules file to use. Default is T4RULES.LIS. See below for
details of rules file format.
/stack
For Overlay or Multifile charts, stack each parameter.
/timerange=range
Only graph the records in the specified time range.
The range should be specified as /t=hh:mm,hh:mm or, if the records
are not from the same day, use /t=dd-mmm-yyyy:hh:mm,dd-mmm-yyyy:hh:mm
/u[=htmfile]
Just generates statistics listing. No charts are generated unless a
filespec is provided. If a filespec is provide, the filename is also
used for the naming of the .PNG files, e.g. <paramnum>_<filename>.png.
/version.
Shows the version of CSVPNG
/width=value
Specifies the chart width in pixels.
/x[=file][:n]
Extract. Same as /extract but exits immediately without creating graphs.
See /extract for details.
You cannot use /a=n with /x=file:n
/y[=file]
Writes a list of specified parameter names to specified file.
/zero[=value]
Zero base. Make the charts zero based. This is the default for overlay
charts and multifile charts.
If a value is supplied, the Y-scale will be from 0 to the specified value.
If no parameter or qualifier is used, you are prompted for a CSV file and
size of chart required. This is intended to be a 'novice' mode. To avail of all
the possible options, specify the commands on the command line.
A number of CSV files may be supplied on the command line. Wildcard file
specifications can be used, e.g. T4_ALP*.CSV. If more than one CSV file is
specified, the charts for all systems will be overlaid. The CSV files ideally
should be from the same time period and from the same cluster but CSVPNG
attempts to match parameters from all supplied files. The CSV files can
be specified in a file with file type .LIS. e,g, to graph parameters from 6
cluster nodes, create a file, CSV.LIS, with the names of the six T4 CSV files,
(one per line) and use CSV.LIS as the file parameter for CSVPNG.
Note that the parameters from the first CSV file specified are used for the
parameters to chart. e.g. if file1 has 100 parameters and file2 has 120
parameters only the 100 parameters from file1 are used.
Default values for some qualifiers can be specified in CSVPNG.CFG.
Allowable entries in CSVPNG.CFG:
average[=n]
height=<nnn>
width=<nnn>
lines
border[=n]
dots[=n]
zero
k="default caption"
ksize=<x> ; x can be any of T, S, M, L, G
kcolor=<y> ; y can be any of R, G, B, Y, U
lines starting with ';' in CSVPNG.CFG are ignored
Some example CSVPNG commands:
Examples:
CSVPNG /p=all /w=600 /h=300 /l /b T4_MCSILO_24JUN2002_1155_2000.CSV
generates a chart 600 x 300 pixels, with grid lines and border. All parameters
in the file are graphed.
'/p=0' or 'p=all' will chart all parameters
'/p=cache' will chart all parameters containing 'cache' (not case sensitive)
'/p=[MON.DISK]' will chart all parameters containing '[MON.DISK]'
'/p="cpu 00" ' will chart all parameters containing 'cpu 00'
'/p=-[MON.DISK]' will chart all parameters except those containing '[MON.DISK]'
'/p=5,6,7,8' will chart parameters 5, 6, 7, and 8
History File:
When CSVPNG is run it checks to see if a history file for that
day exists. The format is CSVPNG_ddmmyyyy.HIS. If it does not exist, it is
created and every CSVPNG command is added to the .HIS file prefixed with a
number, starting at 1 and incrementing for every command. To rerun a command
from the history file type CSVPNG n, where n is a number in the history file.
Lines which do not start with a number are ignored, so the history file may be
edited to insert comments. The comment lines must not start with a number.
/i suppresses this feature and does not use the history file.
Calculated Parameters:
CSVPNG can do some 'column arithmetic'. For example,
columns can be added or subtracted to form a new parameter. To use this feature,
create a .LIS file containing parameter names and use this format to calculate
new parameters (either ',' or '|' may be used as a delimiter between fields):
[CALC], <param 1>, <op1>, <param 2>, <op2>, <param 3>,...,<new parameter name>
<param 'n'> should be a column name (or partial name) from the CSV file or
may be a constant value. The special codes, $CPUS, $MEMORY, may be used for
the number of Cpus in the system and the physical memory.
<op'n'> must be one of: + - / *
<new parameter name> is the name of the new parameter to be created.
The key words, $CPUS and $MEMORY, can be used in the [CALC] line to refer to
the Number of Cpus in the system and the total physical memory.
Up to 5 parameters and 4 operators may be used.
For example:
create a file CALC.LIS
[CALC], 100, -, Cpu 00 Idle, Calculated Cpu 00 Busy
[CALC], $MEMORY, -, Free List Size, Memory Utilization
and specify the .LIS with the /p qualifier.
A calculated parameter may be used in a later [CALC] line.
Calculated parameters will be written to the new CSV file generated by /e or
/x
T4 Rules:
CSVPNG can be used to trigger rules based on T4 data. To use this feature,
create a file using the following guidelines:
Lines starting with ';' are ignored. Use '|' or ',' to separate each field.
Each line should have this format:
Type 1 Rule:
Field 1 | Field 2 | Field 3 | Field 4 | Field 5
Param. name|operator |threshold|number of minutes to test|Caption for Chart
or
Type 2 Rule:
Field 1 | Field 2 | Field 3 | Field 4 | Field 5
Param. name|operator |threshold|either Avg, Min, Max |Caption for Chart
Field 1 should contain a parameter name or partial name (case insensitive)
Field 2 must be one of the operators '<' '>' or '='.
Field 3 must be a number: the threshold value to test for.
Field 4 can be a number (Type 1 Rule) or 'Avg', 'Max', 'Min', case insensitive
(Type 2 rule).
For a Type 1 rule, the rule is fired if the threshold value is exceeded for the
specified number of minutes (period).
For a Type 2 rule, the Average, Maximum, or Minimum value of the parameter is
tested against the threshold value.
Field 5 is the caption you want displayed on the chart if the rule fires. You
can use the keywords $P1 and $P2 as placeholders for the Threshold and Period
of a type 1 rule.
Use the option /r=<filename) to specify the Rules file to use. If no file is
specified with /r, the default file used is T4RULES.LIS.
When a rule fires, the chart is drawn showing the threshold line and, for Type
1 rules, where the threshold was exceeded.
An additional HTML file is generated called T4RULES.HTM showing only the fired
charts. Also, the fired parameter numbers are written to a file T4RULESPAR.LIS.
Some examples of entries in T4RULES.LIS:
Cpu 00 Idle|<|20|5|CPU 00 Idle % was less than 20% for more than 5 minutes
split|>|3|A|Average Split Transfer Rate average was greater than 3
como| >| 1| a| On average more than 1 process in COMO state
--------------------------------------------------------------------------------
The format of a T4 CSV file can be:
Format 1:
The first four lines are header data:
line1: Header information containing T4 revision info and system information.
line2: Collection date (optional line)
line3: Start time (optional line)
line4: Parameter Headings (comma separated).
The remaining lines are the comma separated values. The first column is the
sample time. Each line represents a sample, typically 60 seconds apart.
or
Format 2:
line1: Header information containing T4 revision info and system information.
line2: $$$ START COLUMN HEADERS $$$
line3: parameter headings (comma separated)
.
.
.
line'n': $$$ END COLUMN HEADERS $$$
Here is a simple example of a format 1 T4 CSV file:
EMPIRE, T4EXTR Version: T4 Version : V3.2-15 ... 01-Oct-2002,OpenVMS V7.2-2
7-NOV-2002, 7-NOV-2002, 7-NOV-2002,
10:43, 10:43, 10:43
[MON]Sample Time,[MON.SYST]Direct I/O Rate,[MON.PAGE]Free List Size
7-NOV-2002 10:44:00.16,81,2121
7-NOV-2002 10:45:00.19,79,2122
7-NOV-2002 10:46:00.22,19,2123
7-NOV-2002 10:47:00.26,3,2124
7-NOV-2002 10:48:00.29,0,2125
7-NOV-2002 10:49:00.30,0,2126
7-NOV-2002 10:50:00.32,20,2137
Feedback and comments are welcome!
pat.moran@hp.com
- Next message: JF Mezei: "Module CREDEF inconsistency between DEC-C and Macro"
- Previous message: Paul Sture: "Re: Micro$oft warns of undetectable spyware security risk ..."
- In reply to: David J Dachtera: "Re: Need performance help"
- Next in thread: David J Dachtera: "Re: Need performance help"
- Reply: David J Dachtera: "Re: Need performance help"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
