On Adopting Major Software Releases and Negative Bugs Metrics

If you look at the timeline for the release of a product (open source or not) it generally forms a power law distribution on euclidean plane similar to the following:


The y-axis is the number of pending critical bugs and the x-axis is time.

At some point the product managers realize that the number of reported bugs are falling in severity and that the total number of bugs is asymptotic to zero anyway so they just pick a release date and go for it.

However, this isn’t the metric I care about.

I don’t have time to screw around with unproven software. There’s a major difference between a 1.0 release with zarro boogs and proven software that’s been pounded on by hundreds of thousands of users for 12 months and had all the kinks ironed out.

This is what I want to use. I don’t want to use the latest and greatest software if I can avoid it… The best developers in the world can NOT simulate all the QA that goes into a release which has been deployed in the wild for months and years.

We’re still on MySQL 4.1 for the majority of our cluster. We have 5.1 deployed in a small role but only because we really needed partitioning.

So I’d like to introduce the concept of negative bugs.

When a complex piece of software reaches 1.0 it doesn’t have zero bugs. They’re there, they just haven’t been reported yet. They’re essentially ‘negative bugs’ because they exist and will eventually be applied but just haven’t been found yet.

In fact, I’m willing to predict that they have a power law distribution similar to the above graph.

I’m going to put my money where my mouth is here as well. I’m going to pay a consultant to look at the critical bug report rate for MySQL 4.1 over time. Number of critical bugs fixed per week should be a good metric. (Note to self – this is what an intern would be good for).

It’s not exactly hard work to find these numbers. The data is public – it’s just tedious work.

If my theory is correct, one could compute a sweet spot at which point you should upgrade to future releases.

For example, say MySQL 4.1 stabilized after 8 months, and MySQL 5.0 stabilized at 7 months. Then waiting 6 months or so to adopt MySQL 5.1 seems like a safe bet.

Update: I just posted a gig on Craigslist for someone to compute the numbers.

  1. Kevin,

    Why do you pick a powerlaw? I don’t buy it. Firstly, when the project starts, there are zero bugs. Secondly, bugs will be introduced in non-linear ways throughout the project.

  2. Yeah. I should have modeled it since the project inception.

    I was really focusing on the taper off portion of the product prior to release.

    Though, I think I can unify both the beta and public period with the same hypothesis so I’ll blog about that tomorrow.


%d bloggers like this: