Re: Regression testing (was Re: Performance issue)

From: Jonathan Noack (noackjr_at_alumni.rice.edu)
Date: 05/10/05

  • Next message: Petri Helenius: "Re: Regression testing (was Re: Performance issue)" 
    Date: Tue, 10 May 2005 13:35:40 -0500
To: Bakul Shah <bakul@BitBlocks.com>

    On 5/10/2005 10:18 AM, Bakul Shah wrote:
    > This thread makes me wonder if there is value in runing
    > performance tests on a regular basis. This would give an
    > early warning of any peformance loss and can be a useful
    > forensic tool (one can pinpoint when some performance curve
    > changed discontinuously even though at the time of change it
    > may be too small to be noticed). Over a period of time
    > one can gain a view of how the performance evolves.
    >
    > This would not be a single metric but a set of low and high
    > level measures: such as syscall overhead, interrupt overhead,
    > specific h/w devices, disk and fs performance for various
    > filesystems and file sizes, networking data and pkt
    > throughput, routing performance, VM, other subsystems, effect
    > of SMP, various threading libraries, scaling with number of
    > users/programs/cpus/memory, typical applications under normal
    > and stressed loads, compile time for the system and kernel
    > etc. etc. etc.
    >
    > The setup would allow for easy addition of new benchmarks
    > (the only way anything like this can be bootstrapped). Of
    > course, one would need to record disk/processor/memory speed
    > and capacities + kernel config options, system build tools
    > and their options to interpret the results as best as
    > possible. For the results to be useful the setup has to
    > remain as stable as possible for a long time.
    >
    > [While I am dreaming...] A follow on project would be to
    > create visualization tools -- mainly graphing and comparing
    > graphs. It would be neat if one can click on a performance
    > graph to zoom in or see commits made during some selected
    > period.
    >
    > Such a detailed look, combined with profiling can help people
    > focus on specific hotspots & feel good about any improvements
    > they are making. This can be a great way to rope in new
    > people;-)

    Sounds great! When do you begin? ;-)

    This has been proposed before and has been (to my knowledge) universally
    accepted as a Good Idea. If you have the interest and time to devote to
    it, I would urge you to work on it. The benefit to the community would
    be huge. 

    -- 
Jonathan Noack | noackjr@alumni.rice.edu | OpenPGP: 0x991D8195

  • Next message: Petri Helenius: "Re: Regression testing (was Re: Performance issue)"

    Relevant Pages

    • Regression testing (was Re: Performance issue)
      ... such as syscall overhead, interrupt overhead, ... The setup would allow for easy addition of new benchmarks ... and capacities + kernel config options, ... graphs. ...
      (freebsd-performance)
    • Re: Regression testing (was Re: Performance issue)
      ... such as syscall overhead, interrupt overhead, ... >and capacities + kernel config options, ... For the results to be useful the setup has to ... >graphs. ...
      (freebsd-performance)
    • Re: Performance Tracker project update
      ... I now have a small setup of a server and a slave chugging along, ... It's neat that you've already been able to spot changes. ... A couple suggestions for the graphs. ...
      (freebsd-performance)
    • Re: Message pops up when I edit graph
      ... > I'm creating graphs in powerpoint and every time I open a graph to edit it - a box ... Please run Setup ... in maintenance mode and install at least one character." ... Featured Presenter, PowerPoint Live 2004 ...
      (microsoft.public.powerpoint)