Four paradoxes of software development

Civil engineers can rightfully claim that no two bridges are exactly alike. However, bridges share many known characteristics, and the materials they are built with have known characteristics. Building bridges involves many known knowns and not as many unknown unknowns as one might think.

I am not a civil engineer, and I have nothing but respect for the fine folks who design and build our bridges, but I point this out to contrast bridge building to writing software. Writing good, functioning software is hard. No project undertaken by software development teams has ever been done before. There are similarities among projects, sure, but every software project has its own nuances, requirements, and plentiful supply of unknown unknowns. 

Or, one might say, software development is full of paradoxes that are challenging to deal with. Here are four.

No one knows how long the job will take, but the customer demands a completion date.

This, frankly, is probably the biggest challenge that software development organizations face. We simply can’t be certain how long any project will take. Sure, we can estimate, but we are almost always wildly off. Sometimes we drastically overestimate the time required, but usually we drastically underestimate it. 

For our customers, this is both a mystery and a huge pain. Not understanding the first part of the paradox, they don’t understand why they can’t know for sure when their new software will arrive. Then of course they are frustrated when the software isn’t delivered as promised. 

We try story points and planning poker and all kinds of other agile techniques to figure out when things will get done, but we never seem to be able to get past Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.

Adding developers to a late project makes it later. 

Known as Brooks’s Law, this rule may be the strangest of the paradoxes to the casual observer. Normally, if you realize that you aren’t going to make the deadline for filing your monthly quota of filling toothpaste tubes, you can put more toothpaste tube fillers on the job and make the date. If you want to double the number of houses that you build in a given year, you can usually double the inputs—labor and materials—and get twice as many houses, give or take a few. 

However, as Fred Brooks showed in his book The Mythical Man Month, “adding manpower to a late software project makes it later.” That is a paradox, but it’s as close to a law in software development as we will get. Brooks showed that because new team members require time to learn the context of a complex system and increase the communication overhead, they can’t contribute to the project immediately, thus lengthening the time to project completion while also driving up costs. 

The better you get at coding, the less coding you do.

It takes many years to gain experience as a software developer. Learning the right way to code, the right way to design, and all of the rules and subtleties of writing clean, maintainable software doesn’t happen overnight. 

But all too often, as you gain that experience, you are put into leadership positions that actually reduce the amount of code that you write. Instead of coding, you end up in design meetings, reviewing code written by others, and managing people. Sometimes you get promoted out of writing code all together. 

That is not to say that a senior developer’s contribution decreases. By planning projects, mentoring younger developers, enforcing coding standards, and realizing how important it is for everyone to write good code, a senior developer contributes mightily to the success of the team and the company.

But you still end up writing less code.

Software development platforms and tools keep getting better, but software takes just as long to develop and run.

If you compare how we build web applications today, with amazingly powerful tools like React, Astro, and Next.js, versus how we built websites 30 years ago, when we processed data and HTML using the Common Gateway Interface (CGI), you realize that we’ve advanced light-years from those early days. 

And yet, while our tools get more and more sophisticated, and our processors get faster and faster, software development never seems to move any faster. Work always seems to expand to exceed not only time budgets, but every CPU cycle as well.

Our sites look nicer, but are we really any more productive? Do our sites run faster and process data better? Sure, these new frameworks and libraries abstract away many complexities (does anyone want to write jQuery code anymore?), but they also introduce new problems like long build pipelines, configuration nightmares, and dependency bloat. 

The existence of these paradoxes doesn’t mean things are hopeless. I don’t point them out to frustrate or depress you. Every day, paradoxically, teams still build and ship working software. 

I point them out to make sure we realize that they exist, that we need to accept them and deal with them, and hopefully avoid the pitfalls and potholes that they present. We can’t eliminate the strangeness and chaos, but we can anticipate them and deal with them. Our job is to ship despite them.

One last paradox might be that software is never really done. There is always one more feature that you can add. At least with a bridge, it is quite clear when the job is finished and that the product works as designed.  Civil engineers can rightfully claim that notwo bridges are exactly alike.  However,bridges share many known characteristics, and the materials they are built withhave known characteristics.  What they dohas many known knowns and not as many unknown unknowns as one might think.I am not a civil engineer and I have nothing but respect for the fine folksthat design and build our bridges, but I point this out to contrast it towriting software.  Writing good,functioning software is hard. Every project undertaken by software developmentteams has never been done before.  Thereare similarities amongst projects, sure, but any given project has nuances,requirements, and a plentiful supply of unknown unknowns.   Or, one might say, software development isfull of paradoxes that are challenging to deal with.  Here are four:No one knows how long anythingwill take.  Customers want and need toknow how long things will take. This, frankly, is probably the biggest challenge that software developmentorganizations face.  We simplyaren’t able to tell for sure how long any project will take. Sure, we canestimate, but we are almost always wildly off — sometimes drasticallyoverestimating, but most often drastically underestimating how long somethingwill take.   But for our customers, this is both a mysteryand a difficulty.  Not understanding thefirst part of the paradox, they don’t get why they can’t know for sure whenthey will have a new piece of functionality and are of course frustrated whenit isn’t delivered as promised.   We try story points and planning poker and allkinds of other agile techniques to figure out when things will get done, but wenever quite seem to be able to get past Hofstadter’s Law: “It always takes longer than you expect, even when you take intoaccount Hofstadter’s Law.”Brooks’ Law — Adding developersto a late project makes it later.  This is the strangest of the paradoxes to thecasual observer.  Normally, if you realize that you aren’t going to make the deadline for filingyour monthly quota of filling toothpaste tubes, you can put more toothpastetube fillers on the job and make the date. If you want to double the number of houses that you build in a givenyear, you can usually double the inputs — labor and materials — and get twiceas many houses, give or take.   However, as Fred Brooks showed in his book The Mythical Man Month, “adding manpowerto a late software project makes it later.” That is a paradox, but it’s as close to a law in software development aswe will get.  Brooks showed that becausenew team members require training, time to learn the context of a complexsystem and increase the communication overhead, they can’t contribute to theproject immediately, thus driving up costs. The better you get at coding, theless code you end up writingIt takes many years to gain experience as asoftware developer.  Learning the rightway to code, the right way to design, and all the small subtleties of writingclean, maintainable software isn’t done overnight.   But all too often, as you gain thatexperience, you are put into leadership positions that actually reduce theamount of code that you write.  Instead,you end up in design meetings, reviewing code written by others, and managingpeople.  And sometimes you get promotedout of writing code all together.   That is not to say that a senior developer’scontribution decreases — that is usually not the case.  The process of planning projects, mentoringyounger developers, enforcing coding standards, and realizing how important itis for everyone to write good code — all contribute mightily to the success ofa project.  But you still end up writing less code. Software development frameworksand tooling keep getting better and more powerful, but our software still takesjust as long to develop and never seems to run any faster.  If you compare how we build web applicationstoday with React,Astro, Next.js, andother powerful advanced tools with thirty years ago when we processed data andHTML using the Common Gateway Interface (CGI), you soonrealize that we’ve advanced lightyear from those early days.   It always seems like a paradox to me that ourprocessors get faster and faster, but software development never seems to moveany faster.  Work always seems to expandto fill and exceed not only time budgets, but every single CPU cycle as well. Our sites look nicer, but are we really anymore productive?  Do our sites run fasterand process data better?  Sure, these newframeworks and libraries abstract away many complexities (does anyone want towrite jQuery code anymore?)  but at thesame time introduce new problems like long build pipelines, configurationnightmares, and dependency bloat.     The existence of these paradoxes doesn’t meanthings are hopeless. I don’t point them out to frustrate or depress you.  And yet, every day, teams still build andship working software.   I point them out to make sure we realize thatthey exist, that we need to accept them and deal with them, and hopefully avoidthe pitfalls and potholes that they present. We can’t eliminate the strangenessand chaos that they can present to us, but we can anticipate them and deal withthem.  Our job is to ship despite them. One last paradox might be that software isnever really done. There is always one more feature that you can add.  At least with a bridge, it is quite clearwhen it is done and that it works as designed.