Why we get buggy software

The manufacturing of physical objects is largely understood, mostly because the same thing is being produced over and over. If you are cranking out widgets, the process is orderly, repeatable, and possibly even perfectable. You can find and improve the worst bottleneck again and again until the whole process is highly efficient. 

Building bridges and houses is less repeatable, but in both cases the process has been developed over many decades, even centuries. There are few surprises.

But building software? That is a whole different cup of noodles. Each project, large and small, is unique and has never been done before. There are any number of ways to solve a given problem and smart people disagree on what is the best way. Maybe there isn’t even a single best way. Often, a better way only reveals itself after you’ve done the project the first time. Once you choose a path, there will always be an unknown number of unknown unknowns. (See what I did there?)

Sure, we have developed some best practices over the years and some patterns have emerged, but developers will even differ over how those patterns might be implemented. 

In other words, there is generally a pretty clear best way to build widgets, but there are a seemingly infinite number of ways to build a software project, making the notion of a “best way” problematic.

Crystal ball in a black hole

And this, of course, makes figuring out how long a software project will take very, very hard. Even after years of trying, no one seems to have found a reliable way to do it. Many folks — me included — consider it pretty much impossible.

Now the phrase “pretty much impossible” is not one that a business owner wants to hear. If you are producing and selling software, your customers do not want to hear you say, “You know that software you asked us to build? Well, we aren’t at all sure when it will be done. We are still figuring out the right way to build it. We tried one way, and that didn’t work out, but we might have figured out a better way as a result. But because we aren’t completely sure how we are going to do it, we can’t tell you when it will be ready and working.”

Now, while the hypothetical confession above is the truth, few software development shops will admit that to the customer, much less to themselves. They might not even realize it is true. 

The software industry has gone (and continues to go) to great lengths to solve this problem, and many a product marketing manager for software process tools will tell you they have solved it. (Trust me, they haven’t.) Much marketing copy has been written, and many promises have been made, but the fact remains: No one knows when that software project will get done.

As a result, an inevitable tension exists between the groups that are trying to produce new software and features and the groups that are trying to make money with these new products and features. 

Because software is so malleable and able to deliver new features, customers make purchasing decisions partly on current features, but often on “the next big thing.” As a result, the business side promotes, and often even sells, features that don’t yet exist. Naturally, customers want to know when these features will arrive. And naturally, software businesses promise to deliver features within a certain time frame. 

Very often, those promises are hard to keep. Software development teams would love to be able to tell the business folks precisely when things will arrive, but, as noted above, they can’t. (They might say and act like they know, but they don’t know….) They are often pressured into giving a date. Much effort is put into predicting that date, but those estimates are almost always wrong. Sometimes they are spectacularly wrong.

The two-legged three-legged stool

So what is a development team to do? There are dials that can be turned. The first dial everyone turns is to work more hours. That might work. However, pressuring a team usually results in more shortcuts and more errors, increasing technical debt but not necessarily producing an on-time delivery. Managers are tempted to add more people to increase the hours worked, but then Brooks Law kicks in and things get worse. 

Altering the delivered feature set is another common solution. Another path is sacrificing quality, whether via reducing the bug fixing or decreasing the effectiveness of the user interface. Finally, one can move the schedule, but that irritates paying customers.

Hence we have the famous three-legged stool of “quality, schedule, or features — pick any two!” The business hates to sacrifice schedule, and sometimes will flat refuse to do it. Customers, whose feelings are dreadfully important, don’t like shifting dates and generally abhor scaling back features. 

That leaves quality on the chopping block, and let’s be honest, this is usually the path taken. No one likes a glitchy application, but it is both easy and tempting to hide poor quality behind apparently working features. It is all too routine for customers to take delivery of “working software” and only discover the lack of quality after the fact. The software team then fixes the bugs and delivers a “point release” that improves quality. This is, in effect, a very sly way to alter the schedule — put the bug-fixing time after the delivery time. 

And that “clever” way of solving the problem is normally the one chosen. The development team gets to (supposedly) hit the date, the features promised to the customer are delivered, and everyone is happy. Until they run the software. 

We aren’t making gadgets and gizmos. In the end, someone has to eat the turd sandwich, and it usually ends up being the software developers and the QA team. And that is why you get buggy software.