I wrote the text below for a client and I thought it might be useful for others to use. I'm releasing under the MIT license.

A Primer on Software Development and Its Frustrations

If you are ever working with clients who are new to software development this can help properly set expectations and get everyone on the same page. Feel free to edit as you wish and send PRs if you think others would benefit from your changes.

It’s an open secret among software developers that there is some mystical entropy that infects all projects. Unlike other hiring situations – like hiring a plumber to fix a leak – hiring someone to build software has too many variables to contend with.

It's not pipes and fittings and building codes – it’s words and colors and interactions. It's complex systems talking to other complex systems, possibly ones with which the developer has no prior experience. All to please the moving target of business needs and human desires. It’s a mix of customer service, art, and engineering – all wrapped up in one.

The job is done not when the water stops leaking, but when the client is satisfied enough with the state of the open-ended project that they declare a success, or at least enough success for now.

Most people who hire plumbers have tried to fix the leak themselves or have had some sort of first-hand experience with the problem in question. Very few purchasers of websites and software have attempted to do what they’re asking the developer to do, and that lack of experience is a big source of suspicion and doubt. There’s no visceral way for the client to evaluate a developer’s performance.

Another thing that might not be clear to the inexperienced: It’s impossible for the developer to account for all the variables before development. There are several reasons for this. For one, the people and business needs the project is supposed to serve continue to change while the development is happening. And unlike a building, the plans are 'finalized' as the software is built, not before hand. It's the dark side of software's instantaneous and potentially limitless nature. There are no real physical constraints to rein in a project.

Also, desires change not only in parallel to development, but in response it. Problems are uncovered. Problems are created by accident. You might as well try to model reality itself. At the beginning of the project, the client (understandably) wants the developer to see all the pieces and put them together in her mind, and then tell the client if there will be any problem. That’s essentially telling the developer to completely build the website in her head before she even gives an estimate.

This state of things just doesn’t seem right, to either side. And it’s hard to point to the exact source of this entropy because, like climate change, it’s roots are complex, hard to define, and it creeps in too slowly for many people to notice.

When I was a more junior developer I thought that people were using this uncertainty as an excuse to not work hard enough or to cover up their own ineptitude. It took dozens of projects going over budget and off the rails – despite great developers working on them – for me to believe it.

Developers themselves are to blame for much of this expectation mismatch. We tend to think that we can do things super quickly and give outsiders the false sense that we are 'wizards'. This is partly because we like the awe and respect it brings us, but more so because of the lack of ability to model the whole project in our mind ahead of time. We genuinely believe that things will be easy, no matter how often we've been burned.

So unless the developer you're talking to is the developer that will ultimately be saddled with the burden of your project, and even if they are, to be honest, take what they say with a giant grain of salt. They don't mean to mislead you, and they don't think they are, but chances are they're giving you a rosier picture of your problem and requirements than is realistic.

Why Did This Other Developer Say They Could Do It In Three Months For Half the Price?

Software developers confidently give concrete estimates in order to get business. It’s hard to plan a business around uncertainty, so if one developer is telling you it’ll be two months and another is saying they don’t know how long it will be, the second one seems unreliable and unappealing even though she is being truthful and the other one is lying.

Once the developer gets her foot in the door, the project almost always goes over budget and over the deadline. While it's likely to happen, each time it happens it's usually for unique reasons and not clearly the fault of the client or developer, although each side will blame the other. Quality developers are a necessary but not sufficient condition for success. Success depends more on the framing of the endeavor, the quality of communication, and the matching of expectations than it does the adherence to deadlines and milestones.

The more experienced a client is, the less she demands from the developer in terms of timelines and budget estimates. The smart money has seen how the sausage is made so the it will put more weight on trust, dedication, and track record.

A common scenario is this. An agency needs a website, so they put out an RFP describing what they think they need. Developers all scramble to read between the lines of what the agency is saying and what their expectation of cost is. Then they evaluate whether or not the agency’s expectation is an acceptable amount for the work that the developer thinks is needed. The proposal submitted is more like a poker bet than a bid. Get the client interested enough to stay in the hand so you can increase the pot.

When a proposal is accepted by the client, it’s like an arranged marriage. The two parties don’t even get to meet each other until all the contracts have been written up, which are all based on complete conjecture about whether this will actually be a successful relationship.

What is lost in this process is the problem that actually needs to be solved for the client. To fix this, clients need to become smarter and wiser about software development and understand that the inherent entropy involved means that you can’t use your same judgments to evaluate software contractors as you would for other trades or services.

If you're primarily driven by cutting costs, you'll actually spend more because keeping things on budget and on deadline are directly at odds with properly exploring, defining, and fixing your problem.

If clients become the smart money at the table, they can get more of what they want for less money, instead of being suckered into a drawn out round of betting in which they ultimately pay more and feel swindled – which essentially is what the RFP process is.

A successful software project is one that is viewed as an exploration, rather than a building built to spec. It’s, “We’re trying to get to this idea over here and we trust you as our guide, so let’s move in that direction.” Not, “I know what this needs to be, it's your job to build it.”

But don’t just take my word for it. Lots of smart people have come to the same conclusion:

The Myth of Software Time Estimations

https://hackernoon.com/the-myth-of-software-time-estimations-576a7466d91a

Hourly Billing Is Nuts

https://jonathanstark.com/hbin

Value-based Pricing vs. Cost-based Pricing in Web and App Development

https://www.brainleaf.com/blog/running-a-business/value-based-pricing-vs-cost-based-pricing-in-web-and-app-development/

Cover image by Matias Malka on Unsplash