The Status Quo

There's no "one way" for organizations to build and deploy a product. Even a single engineer will use different techniques in different contexts. When I'm at work, for example, I use a Slack-based chatbot to trigger new deployments; which, in turn, communicates with Kubernetes; which, in turn, executes an incremental rollout of new Docker containers. But, in my personal life—on side projects—I still use FTP in order to manually sync files between my development environment and my production environment.

No given approach to web development is inherently "right" or "wrong". Some approaches do have advantages. But, everything is a matter of nuance; and, every approach is based on some set of calculated trade-offs. At work, I get to use a relatively sophisticated deployment pipeline because I stand on the shoulders of my brilliant teammates. But, when I'm on my own, I don't have the ability to create that level of automation and orchestration.

Though many different approaches exist, most build and deployment strategies do have one thing in common: when code is deployed to a production server, users start to consume it. Add a new item to your navigation and—immediately—users start to click it. Change a database query and—immediately—users start to execute it. Refactor a workflow and—immediately—users start to explore it.

We're decades beyond the days of shipping floppy disks and CD-ROMs; but, most of us still inhabit a state in which deploying code and releasing code are the same thing. Having users come to us (and our servers) allows us to respond to issues rapidly; but, fundamentally, we're still delivering a static product to our customers.

To operate within this limitation, teams will oftentimes commit temporary logic to their control flow in order to negotiate application access. For example, a team may only allow certain parts of an application to be accessed if:

• The user is connecting though the Virtual Private Network.
• The request is coming from a set of allow-listed IP addresses.
• The authenticated user has an internal company email address.
• The incoming request contains a secret HTTP Cookie value.
• The incoming URL contains a special query-string parameter.
• The HTTP User-Agent contains a magic string.

I've used many of these techniques myself. And, they all work; but, they're all subpar. Yes, they allow internal users to inspect a feature prior to its release; but, they offer little else in terms of dynamic functionality. Plus, exposing the gated code to a wider audience requires changing the code and deploying it to production. Which, conversely, means that re-gating the code—such as in the case of a major bug or incident—requires the code to be updated and re-deployed.

These techniques often treat deployment as an afterthought. Meaning, they're typically implemented only after a feature has been completed and is ready for review. This implies that the feature was defined and developed in relative isolation.

This is referred to as the Waterfall Methodology. Waterfall Methodology comprises a set of product development stages that get executed in linear sequence:

1. Analysis and requirements gathering.
2. Graphic design and prototyping.
3. Implementation.
4. Testing and quality assurance (QA).
5. Deployment to production.
6. Maintenance.

Each one of these stages is intended to be completed in turn, with the outputs from one stage acting as the inputs to the next stage.

At first blush, the Waterfall Methodology is attractive because it looks to create structure and predictability. But, this is mostly an illusion. In the best case scenario, the timeline for such a project is grossly underestimated. In the worst case scenario, the engineering team builds the wrong product.

Years ago, I worked at a company that used the Waterfall Methodology. On one particular project for a data-management tool, the team carried out the requirements gathering, did some graphic design, and then entered the implementation phase as per usual. But, building the product took a lot longer than anticipated (which is standard even in the best case scenario). And, naturally, the client was very upset about the slipping release date.

Finally, after many delays and many heated phone calls and much triaging, the team performed their big reveal to the client. And, after walking through the product, the client remarked, "This isn't at all what I asked for."

It turns out, there was a large understanding gap in the requirements gathering phase. This understanding gap was then baked into the design process which was subsequently baked into the engineering process.

The client was furious about the loss of time and money with nothing to show for it. The engineers were furious because they felt that the client hadn't been forthcoming during the analysis phase (classic victim blaming). And, the product managers were furious because the project failed and reflected poorly on the firm.

This isn't a black swan event. Many of us in the product development space have similar stories. And, as an industry, we've come to mostly agree that the Waterfall Methodology is problematic; and, that Agile Methodology is the preferred approach.

The Agile Methodology takes the Waterfall Methodology, shrinks it down in scope, and repeats it over and over again until the work is complete. The cornerstone of an agile development workflow is a strong emphasis on "People over processes and tools" using a continual feedback loop:

• Build small.
• Show your work.
• Refactor based on feedback.
• Repeat.

If our team had been using an agile development workflow (in the earlier anecdote), success would have been achieved. The client would have seen early-on that the product was moving in the wrong direction and they would have told us. In turn, the design and engineering teams would have changed course and adapted to the emergent requirements. And, ultimately, all stakeholders would have been happy with the outcome.

The Agile Methodology is fantastic!

At least, it is within a greenfield project—one in which no prior art exists. But, as soon as you release a product to the customers, all subsequent changes are being made within a brownfield project. This is where things get tricky and expensive.

Agile Methodology has clear advantages over Waterfall Methodology; but, eventually, both approaches run into the same fundamental problem: deploying code to production is still a dangerous proposition. And, even with an agile approach, bad things will happen. And so, the fear creeps in; and soon, in an effort to protect themselves, even agile-minded teams start to fall back into old waterfall tendencies.

In hopes of leaving nothing to chance, the design process becomes endless. An eroding trust in the engineering team leads to a longer, more tedious QA period. Paranoia about outages means no more deploying on Fridays (or, perhaps, with even much less frequency). Test coverage percentage becomes a target. Expensive staging environments are created and immediately fall out-of-sync. A product manager creates a deployment checklist and arbitrarily makes load testing a blocking step.

The whole process becomes heavy and bloated and starts to creak and moan under the pressures—of time, of cost, of expectation—until, at some point, someone makes the joke:

Work would be great if it weren't for all the customers.

In an industry where customer empathy builds the foundation of all great products, wanting to work without customers becomes the sign of something truly toxic: cultural death.

This isn't leadership's fault. Or the fault of the engineers or of the managers or of the designers. This doesn't happen because the wrong technology stack was chosen or the wrong management methodology was applied. It has absolutely nothing to do with your people working in-office or being remote. This happens because a small seed of fear takes root. And then grows and grows and grows until it subsumes the entire organization.

Fear erodes trust. And, without trust we don't feel safe. And, if we don't feel safe, the only motivation that we have left is that of self-preservation.

This sounds dire; and it is; but it isn't without hope. All we have to do is address the underlying fear and everything else will eventually fall into place. This transformation takes time. But, it can be done; and, it starts with feature flags.

Have questions? Let's discuss this chapter: https://bennadel.com/go/4541