Re: Quality of FreeBSD

• Previous message: Robert Watson: "Re: READ_DMA, WRITE_DMA errors"
• In reply to: Alexey Yakimovich: "Quality of FreeBSD"
• Next in thread: Marc Olzheim: "Re: Quality of FreeBSD"
• Reply: Marc Olzheim: "Re: Quality of FreeBSD"
• Reply: Matthias Schuendehuette: "Re: Quality of FreeBSD"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Date: Thu, 21 Jul 2005 12:00:31 +0100 (BST)
To: Alexey Yakimovich <aiy@ferens.net>

On Wed, 20 Jul 2005, Alexey Yakimovich wrote:

> My advice to FreeBSD release engineering team:
> - do more testing;
> - have it tested with hardware what was published in "Hardware Notes";
> - do not release it for production if it is not in production quality;
> - reread again what was written by yourself regarding 4.4 release
> quality.
> I wish to say more.
>
> This mail was written because I like FreeBSD and I want to continue
> using it. And wouldn't mind to wait longer for real production quality
> releases instead of start using something else. And please, I know, it's
> open source project.

While I agree more testing always helps, and that there are some fairly
concete ways we can work to improve testing, there are also some practical
realities to how software testing happens, especially for complex software
products running on diverse hardware. I have a question for you though:

   Have you tried, and do you plan to try, our 6.0 test releases before
   6.0-RELEASE goes out the door? Specifically, on the hardware you know
   you're having problems with 5.4 on?

The way hardware gets tested is that people who have the hardware run the
software on it under a variety of loads, and see if it works. Since a
volunteer project of a couple of hundred developers can't buy all known
past and future hardware, we have to rely on hardware vendors, software
resellers, and FreeBSD users to do some of the testing. In order for that
testing to affect a release, it must happen before the release goes out
the door, rather than afterwards. And it has to happen sufficiently in
advance of the release that someone can do something about the results of
failed testing. If hardware isn't tested before the releasee, then
inevitably people with that untested hardware are more likely to
experience problems. This means that the best way to help us support your
hardware is to run our test releases with useful workloads, and then
provide feedback if/when they don't work. I realize you're providing
feedback now on the 5.x branch, but what you may or may not know is that
in the 6.x branch, we have a significant update to the ATA code that may
get merged to 5.x, if it proves to be as much better as we hope. This
means that we need you to test the future code, not the current code, in
order to fix the problems you are experiencing.

90% of useful FreeBSD testing happens when large FreeBSD consumers take
release of FreeBSD and deploy them in their testbeds and real-world
environments, and find the bugs through the application of high levels of
load and obscure hardware configurations. This is why later FreeBSD
releases along a -STABLE branch are typically much more stable than
earlier ones -- the code has run on millions of machines for untold
amounts of load, instead of the thousand or so with a very selected load
it's likely to run on during development. This is how all software
vendors work, really -- be it Microsoft, or Apple, old-style UNIX vendors,
or any of the Linux vendors. Some set of users sits on the bleeding edge
and shakes out the early problems, and then the rest of the user base
suffers through the later versions to shake out more subtle problems that
gradually get resolved.

The FreeBSD Project is working on moving towards a more formal testing
regimen. This change will help shake out software bugs relating to
workload -- i.e., IP stack bugs, file system bugs, etc. But the chances
of it having a significant impact on broad hardware testing is very low.

So if you have non-production instances of your production hardware, and
can reproduce the workloads of your production environment on that
hardware, what we would love you to do is run 6-CURRENT on it and tell us
if that works better. If it does, then it's a question of back-porting
the functionality (if possible) to 5.x. If it doesn't, then we can fix
the problem in the active development tree, then merge as makes sense.
4.x became a great success after a quite shaking 3.x release branch, and
after some bumps early in 4.x. It got there because of a lot of testing
and improvement resulting from production experience. If you didn't have
problems with 3.x and 4.x, it's because someone else got there first.

The reason I suggest waiting for BETA2 is that BETA2 will have cleaned up
support for running 5.x applications. Specifically, there are one or two
system calls that have changed in 6.x, and require COMPAT_FREEBSD5 to be
compiled into the kernel, which it wasn't in BETA1. Likewise, a number of
library version bumps and compatibility pieces will be in BETA2. This
will make it easier to test 5.x application workloads on a 6.x install.

We take the concerns you've expressed seriously, and you should know that
every FreeBSD developer I've talked with in the last few years has been
talking about how to improve 5.x stability. The challenge has been to
integrate the agressive feature set improvement in 5.x with our stability
goals. Much of that improvement has happened for 6.x, and I think you'll
find that you're much happier with the general level of testing and
support there. This was possible because people running 5.x have provided
us a lot of detailed feedback and bug reporting. 6.x is much less
agressive in terms of feature set, and cleans up many of the architectural
changes that made 5.x such a feature-rich releasee. Your feedback on 6.x
sooner rather than latter will improve the quality of the 6.x release, and
we'd appreciate it greatly if you could help us test it!

Robert N M Watson
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

• Previous message: Robert Watson: "Re: READ_DMA, WRITE_DMA errors"
• In reply to: Alexey Yakimovich: "Quality of FreeBSD"
• Next in thread: Marc Olzheim: "Re: Quality of FreeBSD"
• Reply: Marc Olzheim: "Re: Quality of FreeBSD"
• Reply: Matthias Schuendehuette: "Re: Quality of FreeBSD"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]