Life-Cycle of a Feature Flag

A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: a complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a simple system.

— John Gall (Gall's Law)

If you're used to taking a feature entirely from concept to finished code before ever deploying it to production, it can be hard to understand where to start with feature flags. In fact, your current development practices may be so deeply ingrained that the value-add of feature flags still isn't obvious—I know that I didn't get it at first.

To help illustrate just how wonderfully different feature flags are, I'd like to step through the life-cycle of a single feature flag as it pertains to product development. This way, you can get a sense of how feature flags change your product development workflow; and, why this change unlocks a lot of value.

For this thought experiment, let's assume that we're maintaining a collaborative task management product. Currently, users can create and complete tasks. But, they can only discuss these tasks offline. What we'd like to do is build a simple comment system such that each task can be backed by a persisted conversation.

My goal here isn't to outline a blueprint that you must follow in your own projects. My goal is only to shift your perspective on what is possible.

#Flesh-out Work Tickets

Traditionally, when building a feature, tasks in your ticketing system represent work to be done. And, when using feature flags, this is also true. But, instead of arbitrary milestones, you need to start thinking about each ticket as a deployment opportunity.

Not every ticket is going to represent a one-to-one relationship between code and deployment. But, starting off with this constraint will help you think about deconstructing the work into small meaningful changes. And, ordering those changes in such a way that deployments don't break production.

Consider this list of tickets for our new commenting feature:

• Configure feature flag.
• Add link, route, and placeholder view.
• Add new database table for comments.
• Add command for creating a comment.
• Flesh-out UI for adding a comment.
• Add query for listing comments.
• Flesh-out UI for listing comments.
• Add command for deleting a comment.
• Flesh-out UI for deleting a comment.
• Add command for updating a comment.
• Flesh-out UI for editing a comment.
• Add analytics and tracking.
• Release feature.
• Remove feature flag.

When solving problems for our users, we want to focus on getting to the user experience as soon as possible. A good user experience is ultimately what drives the success of a new feature. And, the sooner we can start iterating on the UX, the more opportunity we have to create a compelling experience. As such, when we consider both the breakdown and the ordering of tasks, we usually want to complete just enough back-end work (commands and queries) to unblock the corresponding front-end work.

If you're used to completing an entire feature before deploying it, this breakdown may seem unnecessarily granular. But, suspend your disbelief for a few minutes; and, I'll show you why taking small, incremental steps creates a lot of safety; and, opens the door to a more dynamic and reactive product development mindset.

Let's explore each ticket in turn.

Note: The code samples in this chapter are going to be heavily truncated. I will do my best to keep them clear and insightful; but, please understand that I'm omitting a lot of code for the sake of brevity.

#Configure Feature Flag

As outlined in the previous chapter, this type of feature flag is what I consider to be a "product" feature flag. Meaning, it's a temporary flag that only exists during our feature development. To keep with my recommended naming convention, let's construct the feature flag using the product- prefix, the ticket number (of our epic, user story, case, etc), and a descriptive suffix:

product-TASKS-103-comments

In most cases, a Boolean-based feature flag is the right choice. Our case is no different. All we need to do here is hide the ingress to a new feature: a simple on/off. And, since we don't have anything to show yet, let's configure our feature flag to be off for everyone:

{
	"product-TASKS-103-comments": {
		variants: [ false, true ],
		distribution: [ 100, 0 ]
	}
}

As you can see, 100% of all requests will be served the false variant for our new feature flag.

#Add Link, Route, and Placeholder View

Now that we have our feature flag configured, we can start building our feature. For the sake of simplicity, let's assume that we have a traditional, multi-page application; and, that our task comments are going to be rendered on a new page within our application.

The smallest meaningful first step that we can take is adding the link that brings the user from the list of tasks over to the comments page (for a given task). This link is the ingress to our new feature; and, will need to be gated behind the feature flag.

To do this, we'll evaluate the feature flag for the requesting user; and then, if the feature flag is enabled, we'll render the link.

<cfscript>

	canSeeComments = features.getVariant(
		"product-TASKS-103-comments",
		{
			key: request.user.id,
			userEmail: request.user.email
		}
	);

</cfscript>
<cfoutput>

	<h1>Tasks</h1>

	<cfloop item="task" array="#tasks#">
		<li>
			#encodeForHtml( task.description )#
			<a>Mark Complete</a>

			<cfif canSeeComments>
				<a>Comments</a>
			</cfif>
		</li>
	</cfloop>

</cfoutput>

As you can see, we're evaluating the feature flag and using the returned variant to dynamically add HTML to the rendered output. Right now, the feature flag is configured to serve false for everyone. But, we know that we'll need to conditionally enable this feature in the near future; so, we're passing-in the userEmail for use in said targeting.

Now that we have our link in place, we need to add the new comments page. For this ticket, all we need is a placeholder page—something to link to; we'll flesh-out the details in a future ticket.

Gating the link (<a>) to our feature is an obvious requirement. But, less obvious is the fact that we need to gate the destination page as well. This serves to block unauthorized access and to provide an emergency shut-off should something critical be exposed on the comments page.

<cfscript>

	canSeeComments = features.getVariant(
		"product-TASKS-103-comments",
		{
			key: request.user.id,
			userEmail: request.user.email
		}
	);

	// Ensure that a user isn't attempting to bypass our
	// feature gate on the task-list page.
	if ( ! canSeeComments ) {

		throw(
			type = "FeatureNotEnabled",
			message = "Task comments not enabled for user."
		);

	}

</cfscript>
<cfoutput>

	<h1>Task Comments</h1>

	<p>Coming soon...</p>
	<a>Back to Tasks</a>

</cfoutput>

As you can see, this placeholder view does nothing more than check the feature flag state and provide a link back to the task list.

At this point, we can ship it to production!

When you're new to feature flags, this first deployment can be very uncomfortable. It feels as though we're about to release something "broken". But, remember, the magic of feature flags is that we've decoupled release from deployment. We're not actually releasing any new functionality to our users—we're simply deploying code to the production servers (where it shall remain dormant until we enable the feature flag).

Once I have this basic gating in place, I start to differentiate between the development environment and the production environment. While the set of feature flags is the same across environments, the targeting within each feature flag is unique to each environment.

To keep the development workflow simple, let's enable the feature flag for all users in the development environment and only enable it for internal users in the production environment.

// DEVELOPMENT environment targeting.
{
	"product-TASKS-103-comments": {
		variants: [ false, true ],
		distribution: [ 0, 100 ]
	}
}

// PRODUCTION environment targeting.
{
	"product-TASKS-103-comments": {
		variants: [ false, true ],
		distribution: [ 100, 0 ],
		rule: {
			operator: "EndsWith",
			input: "userEmail",
			values: [ "@example.com" ],
			distribution: [ 0, 100 ]
		}
	}
}

In our development environment, everyone gets the true variant making it easy for developers to work on the new feature. But, in production, everyone still gets the false variant by default. Only users with an email address that ends in @example.com (our corporate email domain) will get the true variant and will be able to see the gated code.

At this point, we can share the feature with our team!

No plan survives first contact with the enemy.

— Helmuth von Moltke the Elder

If shipping incomplete code to production made you uncomfortable, sharing incomplete code with your teammates might make you downright anxious. We're now stepping into a radically different mindset: one of inclusion rather than isolation.

Let's take a moment and consider the full journey of a product. At first, it's nothing but an idea in our heads. Then, perhaps we do a few rough sketches with pen on paper to help bring those ideas into focus. And then, maybe we create a mock-up, which then gets turned into a prototype. And, eventually, code is written and deployed to production.

At each one of these steps, the fidelity of our idea increases. And, with the increased fidelity comes a richer feedback loop. The link (<a>) that we just added to our task list in production is going to feel different than it did in our prototype; which is going to feel different than it did in our mock-ups. The sooner we can get that production version into the hands of our teammates, the sooner we can collect feedback and evolve the concept using real world evidence.

Whatever idea we originally had for our feature, some portion of it is going to be flawed. It's impossible to design an entirely correct solution within the confines of one's imagination. And, all of our poor assumptions and oversights will come to light when we build and share our materialized vision with other people.

This is why it's so important to start small, iterate, and collect feedback along the way. Instead of being afraid of making mistakes, we must learn to embrace our imperfections; and, lean on both our team and our customers to help identify issues early when the cost of making a change is far lower.

#Add New Database Table for Comments

Now that we have our placeholder view in production, we need to start fleshing-out the commenting experience. And, to do that, we need to create a new database table in which to store our comments.

At this point, the structure of our database table is going to be based on an educated guess as to how we believe the feature is going to work. We know that comments are going to be task-specific; and, that each comment will be posted by a user at a moment in time. And, with that information, we have enough to create something:

CREATE TABLE task_comment (
	// ... details are irrelevant ...
);

And at this point, we can deploy that SQL table to production.

If shipping incomplete code to production made you uncomfortable, and sharing an incomplete feature made you anxious, deploying an unused, unproven database table might feel completely backwards. After all, isn't there a good chance that the structure of this database table will need to change once we start writing our code?

Yes. And, that's OK.

Remember, in a feature-flag-driven development workflow, I want you to think about each ticket as a deployment opportunity that, itself, represents a small, meaningful change to the system. Adding a new database table is meaningful in that it represents a persistence mechanism that now unblocks subsequent persistence-related coding. Our tickets no longer represent completeness, they represent progress.

That said, databases do feel different, don't they? A bit sticky perhaps? Whereas code seems fluid and easy to change, database tables feel much more rigid—much more set in stone. We might even believe that database table structures have to be "correct" before we can deploy them.

But, nothing in product development is ever "correct". Every aspect of our stack—from the code to the database to the platform—is nothing more than a best effort to solve problems with the current information that we have about the demands being placed on the system. As time passes, all of this changes; and, our system must evolve along with those changes. The platform evolves; the code evolves; and, so too must the database.

The reality is, changing our database table structure is no different than changing our code. Yes, there are some additional complexities (such as those related to rolling back changes); but, the sooner we lean into the idea of a fluid, evolutionary database schema, the sooner we can start moving our product forward with confidence.

As a thought experiment, imagine that we get half-way through building this feature and we realize that there's something fundamentally wrong with our table structure and that we have to make a transformative change to it. Remember, customer aren't using it yet—it's still gated behind a feature flag. As such, we can just DROP TABLE and start again. This is an extreme case. But, with feature flags gating access to our feature, even extreme cases like this can be met without concern.

#Add Command for Creating a Comment

Now that we have our initial database structure in place, creating a comment feels like the next smallest meaningful step that we can take in building-out the workflows. Though, creating a comment isn't just one thing. As with many product interactions, there are both front-end and back-end aspects: the front-end, where the user initiates actions; and, the back-end, where said actions are translated into persisted state changes.

If each ticket is meant to unblock subsequent work, we have to implement the server-side portion first. Then, once the server-side logic is ready, the relevant front-end work can be started.

The submission of a request (on the front-end) and the processing of said request (on the back-end) seem so deeply coupled that it can feel strange to work on them separately. But, they handle fundamentally different concerns. Whereas the back-end is all about request routing, data validation, data normalization, state transformation, persistence, and error management, the front-end is all about information architecture, interaction design, affordance, usability, and communication.

Working on only one-half of the interaction at a time allows you to be more focused. This requires a smaller mental model which therefore incurs a smaller cognitive load. We'll talk more about this in the chapter on team dynamics; but, separating front-end work from back-end work also reduces code-review overhead and the mean time to pull request approval.

We've already gated the ingress to our task comments view behind a feature flag. And, depending on how we implement our routing, this may be sufficient. However, if we're exposing our commands through a set of API end-points, we'd want to gate that API access as well.

For the sake of simplicity, let's assume that our task comment CRUD (Create, Read, Update, Delete) operations are being managed in a single API resource. Many web application frameworks have concepts like middleware; or, a before() method—a function that runs prior to any other method invocation within the same resource. This makes it trivial to gate all operations behind a single feature flag check:

// Implementation of our comment resource API.
component {

	// I get called once before each controller action.
	public void function before() {

		var canSeeComments = features.getVariant(
			"product-TASKS-103-comments",
			{
				key: request.user.id,
				userEmail: request.user.email
			}
		);

		// Block any CRUD operations related to task comments
		// if the feature is not enabled.
		if ( ! canSeeComments ) {

			throw(
				type = "FeatureNotEnabled",
				message = "Task comments not enabled for user."
			);

		}

	}

	// ... CREATE command and rest of CRUD methods ...

}

Not only does this before() method now gate our create comment command, it'll naturally gate the future delete and update commands as well.

This feature flag check is almost identical to the one at the top of our task comments view. And, seeing the same code in two different places can often be a "code smell" that indicates a need to refactor and abstract logic.

There's certainly nothing wrong with moving both of these feature flag checks into a centralized location. However, it's important to remember that all of this feature flag logic is temporary by design. "Clean code" is important for long-term maintenance; but, over-thinking intentionally temporary code can be a needless waste of time.

Giving You Permission: When it comes to feature flags, copy-paste-modify is a perfectly acceptable technique.

Once the ability to validate and persist a new comment is implemented, ship it to production! We might need to tweak it later once we start implementing the front-end logic; but, as is the ongoing theme of our work, that's OK. We need to get comfortable with the idea that our product is an evolving concept.

#Flesh-out UI for Adding a Comment

We've already gated access to the task comments page. We've already implemented the back-end logic for processing and persisting comment data. Now, we need to create the front-end HTML form that allows a user to submit a new comment.

Take a moment and just enjoy how narrowly focused our work is becoming? It's no longer a large, overwhelming set of requirements that you build over the course of weeks or months. Each ticket now maps to a straightforward, well-bounded concept that you ship to production within hours or days.

And, every time you complete a ticket and ship code to production, your entire team—from your boss to your project manager to your quality assurance (QA) engineers—gets a true sense of the progress you're making; and, how they can best be setting you and your project up for success.

For example, the moment you ship this form to production, your QA team can start fuzzing the inputs and looking for validation edge-case you haven't considered. And, they can do all of this in parallel with the work you're doing next.

Now, code that HTML form and ship it to production!

#Add Query for Listing Comments

At this point, we have the means to create new comments; but, they're disappearing into the database. As such, the next meaningful change we can make is to start rendering those comments in the task comments view. And, just as with the creation of comments, we're going to break the rendering of comments up into two separate tickets: back-end work and front-end work.

We already have a rough sense about how this page is going to work (it's just a list of comments). As such, we can build the SQL queries even if we don't yet have the UI that renders the data. We must lean into the idea that everything can change as needed; and, that we only have to do the smallest thing we can in order to keep the project moving forward.

Ship those SQL queries to production!

#Flesh-out UI for Listing Comments

The last thing we need to do to round-out the creation of task comments is to render the comments to the screen. We already have our feature flag in place. We already have the data being pulled from the database. Now, it's just a matter of applying the right HTML, CSS, and JavaScript.

And, here's the truly exciting part: once this UI is shipped to production, our team can start using it. And, I don't mean testing it—I mean actually using it in their day-to-day work.

Can they delete comments yet? Nope.

Can they edit comments yet? Nope.

But, the moment our team can create and view comments, there's enough functionality in place to fundamentally change the interaction model for the application. And, the sooner we can get our team consuming this new functionality—and the sooner we can gather meaningful feedback—the more likely we are to build a product that meets real world demand.

#Add Command for Deleting a Comment

There's nothing unique about deleting a comment. We're going to approach this in the same way we approached adding a comment: by building the back-end logic first and shipping it to production.

#Flesh-out UI for Deleting a Comment

And, once the back-end logic is in place, we can add the necessary user interface components for deleting a comment. And, of course, once that's done, we're shipping it to production!

At this point, we have the ability to add a comment and we have the ability to delete a comment. Which allows us to start asking some uncomfortable questions. Like, "is this enough?" Is adding and deleting a comment a sufficiently high bar for this new feature?

Consider other forms of media. For years, we couldn't edit or delete SMS text messages; and yet, people love sending texts. For years, we couldn't edit Tweets; and yet, people love posting on Twitter (now known as "X").

There is precedent for success with a limited set of interactions. And, because feature flags allow us to build and deploy features incrementally, it means that we can start thinking much more critically about our product—and the value that it adds—as it continues to unfold. Our period of reflection is no longer limited by the anachronistic constraint of a fully-developed product.

Plus, we can already base this decision on real world experience. Recall from above that we started using this new feature (internally) once the adding and listing of comments had been deployed. Which means, our team can provide meaningful non-theoretical feedback about how this feature feels and functions in an "incomplete" state. We're not guessing here—we're allowing our understanding of the product to evolve with each deployment. And, we're using this understanding to continually reconsider the product roadmap in real-time.

#Add Command for Updating a Comment

[ON HOLD]: As a company, we've decided that being able to add and delete comments is sufficient for now; and, that we need to stop our current efforts and pivot to other higher-value-add work.

Part of what makes this approach so exciting is that we can now lean on the user to help measure the importance of the omitted functionality. If we go live without the ability to edit a comment and only a handful of users open a support ticket asking for it, it's a strong signal that editing a comment isn't critical to the overall success of the feature. And, that our development efforts can be better spent elsewhere.

But, if our Support team receives a steady flow of tickets asking for edit capabilities, that's a strong indication that we were too aggressive in limiting the feature scope. And, that perhaps we need to pivot back to the commenting project and round-out the rest of the missing functionality.

By allowing the users to guide our development efforts at the edges of the feature, we can engage in evidence-driven development. We don't have to rely on our best guesses or the HiPPO (Highest Paid Person's Opinion)—we can incorporate proven customer demand within our decision making process.

Not only does this help the team prioritize different work streams, it also makes the work more fulfilling. When we can identify user needs and desires in a tangible way, it helps us connect with the work and with the customer. This raises our emotional engagement and makes completing the work feel more satisfying.

And, when a user shares their feedback and then sees that their feedback is taken to heart, they will also feel more connected with the product and the team. They may even develop a sense of co-ownership. In fact, users that help to effect change within a product often become the biggest champions of the product.

#Flesh-out UI for Editing a Comment

[ON HOLD]: See above.

#Add Analytics and Tracking

When building a new feature, it's important to include analytics and tracking so that we have some sense of how the feature is being used. I like to defer this code until just before the general availability release. This gives the feature time to stabilize before we clutter the code with our tracking calls.

Every team is going to handle analytics differently; but, as much as possible, I like to put my analytics tracking in the HTML itself. This allows me to track what the user is actually doing as opposed to some abstracted concept deeper down in the code.

And, once the tracking code is in place, ship it to production!

#Release Feature

The magic of feature flags stems from our ability to decouple release from deployment. We've already deployed code to production several times, but we haven't yet released the feature. At least not to the general audience. In fact, we released our feature to the internal team almost immediately after our first deployment. This allowed us to gather early feedback, tweak requirements, and ultimately modify the scope of the project in real-time.

Now it's time to release our task comments feature to the world. No deployment is need—just a reconfiguration of our feature flag distribution:

// DEVELOPMENT environment targeting.
{
	"product-TASKS-103-comments": {
		variants: [ false, true ],
		distribution: [ 0, 100 ]
	}
}

// PRODUCTION environment targeting.
{
	"product-TASKS-103-comments": {
		variants: [ false, true ],
		distribution: [ 0, 100 ]
	}
}

The development environment was already serving the true variant for all users; and now, so does our production environment.

At this point, we need to start watching the error logs. If there's any hard truth in this world, it's that users always manage to break features in new and exciting ways that we didn't expect! Monitoring a release is always necessary.

In this particular case, we jumped directly from 0% to 100% for the general audience. But, we could have incrementally ramped-up our release over a period of time. Slow releases have the benefit of reducing the blast radius of bugs. But, it also means that the application is operating in a dynamic state; which makes it harder to understand and support.

As a rule, I like to release a feature as quickly as possible. It's absolutely OK to be more cautious when you first start integrating feature flags in your application. But once you become more comfortable with the philosophy, you'll naturally develop a sense of how much caution is merited for a given release.

If you ever find yourself incrementally releasing a feature flag (ie, going from 0% to 100% distribution) over the course of weeks, something is wrong. You either haven't considered what "failure" actually looks like in your feature; and so, you proceed with blind caution. Or, the scope of your work is simply too large; and, you're afraid that anything might break at any moment. Embrace this reluctance as feedback that something in your process needs to be adjusted.

Note: An exaggerated release timeline may also be an indication of unhealthy boundaries (see Ownership Boundaries).

#Remove Feature Flag

Once our feature has been released to the general audience, it's time to clean up the code. But, it's important to put some amount of time in between the release of the feature and the removal of the feature flag. This is to act a safety valve for any unexpected issues (such as a SQL query that suddenly start running slowly under heavy load).

Everyone has a plan until they get punched in the mouth.

— Mike Tyson

When you first start using feature flags, it can be tempting to leave feature flags in place for a long time (if not indefinitely). But, this is fear-based thinking. Feature flags add complexity to the control flow which, in turn, makes the code harder to comprehend and maintain. Feature flags should be considered a necessary evil; and, should be removed at the earliest opportunity.

The amount of time in which I allow a feature flag to remain in the code is a function of the risk that the given feature represents. This isn't an absolute value; but, rather, a combination of the possibility of bugs, the impact of said bugs, and the volume of usage.

For example, if the usage volume is low, I might keep the feature in place for a few days (or weeks) in order to observe a sufficiently diverse set of users. But, if the usage volume is high, I know that bugs will be surfaced quickly. Which means, I need less time to become confident in the correctness of the software (even if there's a lot of risk).

Over time, as you get comfortable with using—and removing—feature flags, this window of observation will shrink naturally. In the beginning, you might leave a feature flag in place for a week or two and roll it out over the course of days. But, after a few months of building features with feature flags—and seeing just how revolutionary they are—this window will likely come down to a single day, with releases taking no more than a few hours.

When it's finally time to remove the feature flag, having a good naming convention will pay dividends. We named our feature flag product-TASKS-103-comments. This makes the relevant gating logic extremely easy to find in our codebase. There's no chance that this text will be used in any other context.

But, imagine if we had named the feature flag something generic like, comments. How confident would you feel about finding and removing all the relevant code? Naming conventions for the win!

#Celebrate

In building this new feature, we did some scary things. We shipped code early; we shared the feature early; and, we started consuming our own work far before it was complete. In doing so—by leaning into that discomfort—we were able to parallelize work across the team, adjust the product in response to early feedback, and, ultimately, re-shape the roadmap even as the product was being developed.

Are you beginning to see the possibilities?

#On Nesting Feature Flags

Unless two feature flags are unrelated, the nesting of feature flags is a significant red flag. The human brain is frail and struggles with combinatory systems. When two feature flags are nested, it creates too many potential execution paths through the code making it hard to reason about what the code is actually doing.

If you find yourself about to put a nested feature flag in place, stop what you're doing and remove the outer flag first. Then—and only then—proceed with adding the new flag. It's important that each feature flag's life-cycle be processed in turn so that the overall complexity of the system is minimized.

Have questions? Let's discuss this chapter: https://bennadel.com/go/4547