Life Without Automated Testing

People who say it cannot be done should not interrupt those who are doing it.

— George Bernard Shaw (misquoted)

This may be shocking to some engineers; but, I haven't written a test in years. And, I deploy code to production anywhere between 5 and 10 times a day.

Now, I'll caveat this by saying that I don't write and distribute library code for broad consumption—I maintain a large SaaS product over which the company has full end-to-end control. If your context is different—if the distance between your code and your consumer is vast—this chapter may not apply to you.

To be clear, I enthusiastically support testing! In fact, I thoroughly test every single change that I make to the code. Only, instead of maintaining a large suite of automated tests that run against the entire codebase, I manually test the code that I change, right after I change it.

Both automated testing and manual testing seek to create "correct" software. But, they accomplish this in two different ways. Automated testing uses tools and scripts to programmatically run tests on your application—without human intervention—comparing actual outcomes against expected results. Manual testing is the process of having a living, breathing, emotional human interact with your application in order to identify bugs, usability issues, and other points of concern that may or may not be expected.

While it's possible to build software without automated testing, manual testing isn't optional. At least, not in the product world. Manual testing is an essential part of the product development life-cycle. You make a change, you try it out, you make sure it works, you see how it feels, you adjust the code as needed.

Building a successful product isn't only about correctness—it's about creating an experience. And, you'll never create a great product unless experiencing the product is a core and consistent part of your product development workflow.

What's true of all bugs seen in production? They passed all the tests.

— Rich Hickey (creator of Clojure)

I'm not here to tell you that manual testing replaces automated testing—this would be a misunderstanding of the role that various testing methodologies play. But, I will say that manual testing is mandatory and automated testing is optional.

And, as with all things optional, it becomes a matter of investment: is the cost of adding and updating automated tests worth the value that they deliver to your team? If your answer is an emphatic, "Yes", then absolutely continue writing your tests. I have no desire to dissuade anyone from using automated tests if they're adding a net value.

But, let's step back and examine why automated tests can be a value-add: they instill confidence. The confidence that you're not about to make a mistake and break production in an unexpected way.

Or, to frame it differently, it's precisely your lack of confidence in the code that acts as the reason to write automated tests. After all, if you had great confidence that the code you wrote is correct, there'd be no justification for spending more time / money / opportunity costs to add yet another layer of confidence.

Feature flags create that type of confidence. In fact, I would argue that feature flags can create more confidence than automated tests. By their nature, automated tests can only test the known knowns within an application. Feature flags, on the other hand, allow us to react to anything—both the known knowns and the unknown unknowns.

When you deploy something behind a feature flag; and when enabling that feature causes an error; you just disable the feature flag. It doesn't matter why the error occurred—if it was a predictable issue or a novel edge-case—it only matters that we can react to the issue quickly and return production to a stable, working condition.

Of course, this confidence won't manifest in isolation. It is a byproduct of the way in which feature flags transform product development. Consider the list of tickets that we closed in an earlier thought-experiment (see Life-Cycle of a Feature Flag):

• Add link, route, and placeholder view.
• Add new database table for comments.
• Add command for creating a comment.
• Flesh-out UI for adding a comment.
• Add query for listing comments.
• Flesh-out UI for listing comments.
• Add command for deleting a comment.
• Flesh-out UI for deleting a comment.
• Add command for updating a comment.
• Flesh-out UI for editing a comment.
• Add analytics and tracking.
• Release feature.
• Remove feature flag.

If I had to work on all of these tickets in my development environment; and, then wait until the feature was completely done before deploying it; and, if deploying it meant releasing the feature to all users at the same time—yeah, I'd be terrified! When working under these constraints, I'd be shocked if my deployment didn't break production.

I used to work this way. My team used to work this way. And—in fact—we broke production so many times that I would often talk about taking down production as being a rite of passage (see Of Outages and Incidents).

But, incidents went from happening several times of day to being a rare occurrence. We didn't make this change by adding lots of tests—we did it by adding lots of feature flags. And, the more we baked feature flags into our workflow, the more stable everything became.

It turns out, confidence has an inverse relationship to the breadth of work being done. Change a lot of things—as we discussed above—and confidence plummets. Change nothing, and you can be infinitely confident that you broke nothing.

Feature flags allow us to get as close as we can to the nothing end of that spectrum while still moving the application forward. By decomposing large efforts into a series of small, independently deployable steps, it keeps the breadth of individual changes very low; which, in turn, keeps our degree of confidence very high.

Small changes also mean small PRs. And, as we'll cover in a future chapter, small PRs beget better reviews. And, better reviews mean that fewer issues ever make it into production.

But, perhaps most importantly, feature flags slowly change the way that we approach application architecture itself. Since feature flags always require code to be added safely and then later removed safely, we're forced to start writing code that is:

1. Easy to build in isolation.
2. Easy to find.
3. Easy to delete.

These three properties necessarily end up creating code that is loosely coupled to the rest of the application. And, this loose coupling ends up being one of the most transformative changes when it comes to writing code with confidence.

Consider what it is that engineers are afraid of when they change code. It usually isn't the obvious knock-on effects—it's the quantum entanglement; the, "spooky action at a distance." They're afraid that when they change this line of code "over here", something "waaaay over there" breaks in an unexpected and non-obvious way.

When you write code using feature flags; and, when you organize code so that it's easy to isolate, find, and delete; this entanglement—this tight coupling—it stops happening. And, when the tight coupling goes away, so too will many bugs. Manual testing then becomes a commensurate activity. Meaning, you test what you changed and no more.

But, this is a transformative way of building software. And, it's hard for many people to understand at first. If I bring up the idea of manual testing to people who love automated testing, they will immediately object:

But then you have to regression test the entire application every time you make any change.

This is incorrect. Or, I should say, this is only correct when the degree of coupling is so high that the blast radius of your changes cannot be predicted. Working in small, incremental steps removes this FUD (Fear, Uncertainty, Doubt); and, makes the outcome of each individual change small, safe, and obvious.

Over time, using feature flags forces your code to become easier to maintain because it forces your team to employ better architectural choices. And, this isn't just for you, it's for everyone. All of the rewards that you get from feature flags are going to be the same for the next developer; and, all the developers after that. Because the safety is baked into the process itself (and, into the architectural patterns that feature flags encourage).

Aside: After maintaining a single product for over a decade, one thing that I've come to understand is that there is a lot of code in your application that never changes. A beautiful side-effect of manual testing is that you never have to worry about testing the code that never changes.

There are, of course, limits to what feature flags can do. Not all changes fit neatly into a feature flag workflow. Which means, some of the broader changes—like updating the version of your front-end framework—will require more extensive regression testing. But, this is a relatively rare occurrence. And, it's an occasional cost that I'm willing to pay.

In the end, no approach is perfect. Everything is a calculated trade-off. Automated testing does have some great advantages (especially when it comes to making sweeping changes within an application). But, feature flags completely changed the product calculus; and now, my team is able to maximize the time that we spend building value-add features for our customers. And, we do so with confidence.

Have questions? Let's discuss this chapter: https://bennadel.com/go/4551