The Curse of Bugzilla

This article isn’t really about Bugzilla, or Jira, or Fogbugz in particular.  These are all fine ways for software managers to manage lists of issues, assign bugs to developers, and track when the bugs get fixed.  It’s really an article about understanding our profession’s miserable success rate at completing new software development projects.  It focuses in particular on what can happen when software development managers get lulled into using issue management databases as their only software engineering tool.   And why, more often than not, they do.

First, let’s get real about software development project management and consider a few important statistics.  Between 70 and 80% of the lifetime cost of any software program is maintenance.  Maintenance is really ongoing development.  There are errors to be fixed, of course, but much more work goes to adding new features, integrating old programs with new ones, adapting to changes in the company, regulatory changes, and so on.  This means that for 80% of the time a programmer will:

1. Install his or her development tools, IDEs, compilers, and so on.
2. Check out a snapshot of the source code.
3. Pick the next bug to work on out of the issue tracker.
4. Do edit/compile/debug cycles until the problem described in the bug database goes away.
5. Check in the code.
6. If it is Friday, write a status report so any new bugs can get added.
7. Go back to step 3.

The manager of this programmer, who I’ll term the “maintenance manager,”  probably hasn’t programmed in years and very likely couldn’t build and run the software  him or her self.  Instead, many (most?) software development managers “manage” by what they imagine is going on, based on weekly status reports and what they read from Powerpoint presentations projected on the wall.  However, all is not lost.  Before his or her own weekly status meeting (in front of, say, some similarly technically challenged directors or veeps) she must:

1. Add any new bugs to Bugzilla.
2. Print out a list of all bugs that closed in the last week and which are scheduled for the next.

Voila!  Instant status report that is correct, unchallengeable, and no one needs to actually understand anything about what is really going on.   You’ve got a perfect list of what’s been finished, what will soon be done, and if someone likes, he or she can extrapolate the ship date based on the number of bugs outstanding.  Say, if each of 10 people can fix an average of 3 bugs a week, and there are 60 bugs left, then the code will be done in 2 weeks.  Better to double that to a month.  Excellent!
This probably does work most of the time.  Chances are our hypothetical development manager started at the bottom as a junior bug fixer, advanced to “can add gnarly new features” developer, and now supervises a crack team of professional bug fixers.  So long as someone else can sell the software, or the internal customer still needs it, they are all made in the shade.

But what if you are part of an outfit trying to write a new program, or some major new feature that is for all intents and purposes a collection of new programs?  Here we have lots of depressing statistics.  First, recall that only 20% of the time will a software development manager be exposed to new development work (the other 80% is maintenance.)  Do a quick Google search on “software project failure rates” and you’ll see that perhaps 70% of software projects are deemed “failures” with about a third getting canceled outright.

Fewer than one in ten software development managers will ever manage significant new software development and succeed.

According to my own informal survey, ten out of ten software development managers claim they are God’s gift at new software development.   Somebody’s wrong. Sorry, but most of these people have done little more than fix bugs and manage others who fix bugs and of the very few who try new development work over half of them will mess it up.  (As an aside, I actually know a handful of managers who I would trust with new software development.  How do you think I learned all this?)

Company after company I have worked for has spent hundreds of millions of dollars on new product development projects that were ultimately scrapped.  The managers responsible for this waste seldom have more than a small idea why they were so “unlucky.”   They see other companies succeed, know that big software projects are theoretically possible, but egos being what they are, these people don’t put the blame where it belongs.  I’ve yet to see one of these people acknowledge that neither their experience nor training prepared them to take on brand new software development.

What does a successfully run “new” software project look like, if it isn’t maintenance?  I suppose you might expect me to offer a somewhat rantish critique of “Agile” development, or Rational’s Unified Process and then bring in the absolute necessity for everyone strive for Software Engineering Institute’s “Capability Maturity Model” level 2, at least.  These are all excellent to learn from and adapt to your particular organization and team.  In fact, I think you should already know something about all of these.  If  you don’t already know about these then please stop reading this, and go check them out.  I know you’ve got an Internet connection or you wouldn’t be reading this. Go!

Hey, welcome back.  From here, I think I can accomplish more by taking the discussion to an even more fundamental level.  Let’s do a little Software Engineering 101 course in 500 words or less.

Boehm’s “Software Engineering Economics” book teaches us that errors left in from the requirements and design phases have costs that escalate by powers of 10 through lifecycle phases.   (Yes, a bug can cost 10,000 times more to fix if you don’t catch it early.)  Fred Brooks’ “Mythical Man Month” teach us that communication costs can overwhelm development teams that grow in size.  We’ve learned that the most productive programmer can be 10 or 20 times more effective than the worst mostly because they can bridge the technical and the problem area (business) domains quickly.

“He who codes first finishes last.”

During what I call new product development, then, you should see:

1. A lot of up front thinking time as requirements are digested and critical architectural decisions are made.  I have observed (as have many of my colleagues over the years) that you want to spend over 1/3 of your development budget on requirements and design before writing any code.  “He who codes first finishes last.”
2. There will be artifacts produced that will persist for the life of the project.  There are usually scenarios (use cases), data models (either UML or database schemas), and descriptions of key algorithms.  The notation is unimportant.
3. Architects design components, and developers negotiate high-level APIs.  These APIs have to be written eventually and it always pays to do them first. The architects are responsible for making sure that all the component interfaces get written, but these are always delegated to the owner of the subsystem.
4. Architecting is an activity that peaks early and drops off.  There are no architects on a project by the time it ships.  It helps to hire architects who can transition into developers by mid-project.
5. Testability is built-in at the level of finer and finer components.  People are not checking in code they haven’t wrung out first.
6. People get feedback on how close their estimates come to reality (and not on how short they are.)  Over time the quality of the estimates will improve and you can control any schedule risks rather than simply sit back and wait for the next crisis.  Unless you like a good crisis.
7. The coding phase is boring.  Any issues that slipped through the earlier design phase are no big deal, and because the code is cohesive, changes are painless to introduce.

If your project is typified by developers working long hours on bug lists containing 10s of thousands of crappy bugs then you have probably botched the job.  You probably wanted to do “Agile”-like development because you heard that the “waterfall” method is bad.  You probably have never even seen up-front design documents before and are probably irritated at me for suggesting that you, who have been doing this job for years, are somehow inadequate.  Most likely your “architects” are the most charismatic developers with the biggest egos who want to make sure no one other than them gets credit for anything (but who only want to work on whatever pleases them.)  Your ship dates have slipped and no one wants to hear that code can’t get tested before it gets checked in, and QA is constantly hammering away at you saying “we’ve got to get the bug count under 100 before we can ship!”  Like I said, if you reach this point you’ve botched the job.

So what does all this have to do with Bugzilla?  (As Arlo Guthrie famously said, “Remember Alice? It’s a song about Alice.”)  Issue trackers are the honey that lure experienced maintenance-only  development managers (and that’s 9 out of 10 of you!) into initially happy, yet ultimate tragic, complacency.  It all starts by using an issue tracker to treat new features as “bugs.”

Usually the feature starts as a large bug that spawns into dozens of bugs as the problem space becomes understood and the work divided.  No one asks for design specs never mind an architecture spec, nor is there any place to put anything like these. (Everything goes into the issue tracker, of course.)  If someone did any up front architecture or design work they’d just get punished for taking too long on his or her bug.

At first, no one knows which parts of the system will be affected, so the people who need to be called in won’t be notified until very late in the game.  This usually happens when one developer (“Suzy”) in department A, tries to check in code “owned” by someone (“Frank”) in department B. At this point Suzy is told in no uncertain terms by Frank’s boss that she is forbidden to modify Frank’s code.  Either Frank gets assigned the bug (unlikely, as there is no way in hell Frank’s boss will take on another bug to make him look bad) or Suzy is ordered to check it in anyways even if it pisses people off.

The quicker the code gets checked in the better, as there are no other metrics for quality other than time.  If the code breaks later, or breaks someone else’s code, or some other department pays to add testing or fix user documentation then this is the other department’s new bug.  By the way, this is time tested technique for one manager to steal budget money from a peer manager and get away with it.  You don’t think this is stealing?  Tell it to the fellow whose annual bonus got dinged for missing ship dates.

Presumably some sort of design work got done on the feature (probably a white-board meeting) but there’s no way to trace back a flurry of bugs whose common root cause was poor design work.

Even better, anyone who likes architecture and design work will get seriously nailed at performance review time because their bug fixes per week number will be low.  How do you think company management chooses who gets laid-off next anyways?  Such folk are better off keeping their heads down low to eke out a few months extra pay before the project folds.

At the end of the day it all boils down to those damn programmers not typing fast enough.  And this is just the way most managers like it, apparently.

Long, long ago in the 1980s, at a software company far, far away in Cambridge, MA, I worked with a fellow I’ll call Jonathan. This fellow led a 20-person team to deliver a major product release.  A highly disciplined process person, Jonathan led his team in a very organized manner.  They delivered on time and on budget, nailed the product requirements, and caused product sales to spike up from $20M to $30M in one quarter.  Fully expecting a fine raise and bonus he instead was scolded and threatened with termination because he “didn’t really do any work.”

No one had to buy pizza for the all-nighters that were supposed to happen.  No VP needed to hold emergency executive committee meetings.  There were only 10 bugs logged and these were each fixed within hours of being reported.  QA finished testing the product in half the budgeted time.  They had nothing to do!  Waste of resources!  With no crises to fix and no record of bugs in the bug database there was no way for Jonathan’s boss to detect never mind measure his value.  This same company later went on to spend $50M (maybe much more) on developing a second-generation product that was never quite finished, ran too slowly to be used, and the company went famously out of business.

Are you starting responsible for a software development project that is less than 50% maintenance? I’d start by interviewing the development managers to see if any of them have new development experience (that was successful) or if any of them can be retrained (salvaged.)  Find out if any of them presided over an orderly (boring) assembly of well designed code that was easy to test, met requirements, and let people go home on time on Friday night.  Find out which development managers have nothing to offer but heroic war stories about “how crazy everything got!” and put them at the bottom of your list.  If all you have are maintenance-only software development managers then you are sunk.  Do you really want to roll the dice when the odds are much worse than 50% that you’ll win?  You’d be better served working with an outside consultant on a job description and hire from outside.

If you find your software development managers have at least some of the right skills, make them all work together to come up with a shared development process.   Don’t be afraid to pay for outside training!  Once everyone thinks through the sort of process that gives the right amount of up-front design, the right emphasis on testability, and the right amount of ceremony (bureaucratic overhead) then everyone on the team will support it.  Every manager will be helping the others to play by the rules.

Honestly, it almost doesn’t matter what the “method” is.  Just taking a little time to have everyone think things through, in advance, can make all the difference.  Remember, “measure twice, cut once!”  Just be sure that whatever this process is, that all software process documents, not just code associated with a numbered bug, are give their due.

And don’t let Bugzilla take over! When new development work and features come up, there will be strong temptation on the part of everyone to start feeding them in as bugs.  Don’t let them!  Force software development managers to keep schedules instead and give feedback to everyone who provides estimates so they can get better at it.  Make sure tasks identify up-front design deliverables and make sure someone actually reviews them and stores them someplace where everyone can get at them.

For those of you who have been burned by the “waterfall” method, or have been bitten by the “agile” bug, don’t think for a second that you can skip the up-front design work.  Recently I’ve been exposed to Rally which is an very fancy bug tracker that lets you track progress in an “agile” style.  For the purposes of this essay, this is just another Bugzilla.  All I’m suggesting is that you add creating and updating these documents to your development cycles.  I’m also presuming everyone knows how to size the work as well.  A paragraph can cover a page of code.

Often a developer will say that he or she doesn’t know enough about what they will be coding to write down a page of spec.  The same person might claim that time is too short for design work and coding can’t wait. This is a load of bull.  If he or she doesn’t understand the problem well enough to write quick design spec then he or she has no business touching any code.

Be patient, however.  You are dealing with a well-meaning maintenance coder, and not someone who knows much about “software engineering” never mind having had any actual (successful) new software development experience.  You’ll need to train this person, walk him or her through writing and reviewing and checking-in the specs (you didn’t think specs were exempt from configuration management, did you?) and make sure the entire organization follows suit  Otherwise this person will feel singled-out for special punishment and the other managers will want to shield their own people from similar abuse.

This is why good software engineering process must come from the top down. Pushing process changes from the bottom is like rolling a big rock uphill.  You can’t do your own job well and as soon as you take your hands off the rock everything goes quickly back to the way it was.

The Curse of Bugzilla is that issue management software enforces terrible development practices and causes brain-rot in software developers and managers who once knew a thing or two about software engineering, good design, and schedule management.  Want to fix your organization’s software process?  Try turning your issue tracker off for a week and see what happens.