Ownership Boundaries

Feature flags are an implementation detail for engineering teams. As such, each feature flag belongs to the engineering team that created it. If we were talking about any other low-level algorithmic decision, this sense of ownership would be self-evident; and, we wouldn't need to have this discussion. However, since feature flags can be both observed and configured outside of the application itself, it's easy for people to misunderstand where the boundaries must be drawn.

#Who Can Manage a Feature Flag

Simple—the engineering team that implemented the feature flag is the only team that can manage the feature flag. Since feature flags are an implementation detail, only the implementing engineers have the necessary understanding of how changes to the configuration will impact the production system. These engineers know which database queries are involved; which API calls are being made; which areas of the application are being touched; and, most importantly, which error logs and performance dashboards need to be monitored during a release.

Your product manager doesn't have this information. Your designer doesn't have this information. Your data scientist doesn't have this information. Your growth engineers don't have this information. Your security and compliance officers don't have this information. No one other than the implementing engineers have a fully integrated understanding of the implications; and, therefore, no one other the implementing engineers should be touching the feature flag.

At the beginning of this book, we talked about feature flags as being little more than dynamic if statements. This code-oriented perspective can help bring the ownership boundaries into focus. The people who have permission to configure a feature flag are the same people that have permission to open your code and edit your if statements. And, if you wouldn't be comfortable with a given person jumping in and editing your code, you shouldn't be comfortable with that person jumping in and editing your feature flag configuration.

The exception to this rule involves open-ended, longer-term feature flags. For example, if a feature flag is being used to gate "T100" features or to act as a make-shift paywall (see Use Cases), targeting of the feature flag can be safely managed by customer-facing employees.

But, this is predicated on the assumption that the feature itself has already been enabled for the internal team and verified by the implementing engineers first. As such, any change that a Sales associate or a Support advocate might make is merely an incremental release on top of an already-enabled feature.

#Who Can Reference a Feature Flag

We operate in a messy, asynchronous world. And, when a company manages several different services, there are usually timing constraints that have to be considered around this asynchronicity. For example, if an upstream service needs to make a breaking change to one of its APIs, this can only be accomplished safely through a series of interlocking, sequential steps:

1. Upstream service exposes the new API with the breaking change.
2. Upstream service deprecates the old API, but keeps it running in production.
3. Downstream service(s) are updated to consume the new API.
4. Once downstream service(s) are no longer consuming the old deprecated API, the upstream service removes the old API from production (leaving only the new API in place).

Any attempt to change both upstream and downstream services at the same time will lead to production problems. This is a matter of physics, not of engineering prowess.

Feature flags do not help with this problem. Or, rather, feature flags should not be used in an effort to escape the aforementioned workflow.

There is a strong temptation to believe that a feature flag, shared across services, can magically remove the pain of keeping services in sync. Do not do this. This creates very tight coupling around a mechanism that is intended to be dynamic and transient.

When two services are tightly coupled, it means that changes in one service will lead to unexpected changes in another service (see Life Without Automated Testing). And, if you attempt to share feature flags across multiple services, this is exactly what will happen. At some point, you'll go to change the feature flag configuration for your service and accidentally break or change the behavior of a completely different service.

If you consider the fact that, at its core, each feature flag contains persisted data in the form of variants and targeting rules, then you can think of the feature flag as being a type of database. A single database—shared by multiple services—is known as an integration database. This approach is widely believed to be an anti-pattern in the programming world due to the tight coupling it creates.

Do not allow your feature flag to become an integration database. Any immediate value that such an approach provides in theory will almost certainly be met with future regret.

More than the inevitable regret, attempting to share feature flags across multiple services becomes a limiting factor in the way that you think about your product landscape. Let's say, for example, that you're adding SSO (Single Sign-On) capabilities to your product suite. This includes adding SSO integrations in:

• Your desktop web app.
• Your mobile web app.
• Your iOS native app.
• Your Android native app.

For product managers that are new to the feature flags mindset, they might ask—or demand—that all four application surface areas be controlled by the same feature flag:

product-AUTH-197-sso-login

Because, in their mind, SSO is a "single thing" that just-so-happens to be spread across four different areas of the product. And so, when the feature flag is enabled, they imagine SSO becoming active everywhere all at once.

But, the unfortunate flip-side to this thinking is that if the feature flag needs to be disabled, then SSO gets deactivated everywhere all at once as well. And, this is probably not the desired outcome. At least, not from the customer's perspective.

To extend this shared-feature-flag thought experiment, imagine that when SSO is enabled, a bug is discovered in the native Android app integration. And, imagine that only 5% of customers use the native Android app. And, of those 5%, only a fraction of customers actually use SSO for their login workflow.

If all login surface areas are controlled by the same feature flag, then disabling SSO for Android customers means disabling SSO for all customers. And, if the vast majority of customers were never experiencing the bug, how can you—as a service provider—justify disabling the feature flag?

Instead, in order to maximize customer value while still enacting safe development practices, multiple feature flags must be used to control the "same feature" across multiple surfaces. Had the concept of SSO been implemented using several different feature flags, you could have disabled the Android native app feature flag and left every other feature flag in place, thereby allowing the rest of your customers to continue receiving the benefits of SSO.

This is what it looks like to think right towards your customers. Feature flags allow you to break the bonds of arbitrary consistency and start thinking about smaller, cohesive ways in which to deliver value to your customers.

But, the boundaries of cohesion are not always clear. And, since every team has a different strategy for both organizing and deploying code, there may not be a one-size-fits-all recommendation.

For most teams, I recommend a repository level boundary. That is to say that all references to a single feature flag should be wholly contained within a single code repository.

This rule needs to remain true from a transitive perspective as well. Which means that a feature flag should never be referenced inside a shared library. If a shared library exists in one repository (where it is developed) and is then pulled into another repository (where it is consumed) then, for all intents and purposes, the same feature flag is being referenced in two different repositories.

Aside: If a shared library needs to present conditional behavior, this should be exposed as an input or configuration property on the shared library. Then, each consuming context must be responsible for providing a mechanism (such as a feature flag) that defines the appropriate input parameter binding.

For teams that use a monorepo—a single version-controlled repository that composes multiple projects—the correct boundary is less clear. And, to be completely transparent, I have very little experience with monorepo style development practices. My best guess, in this case, is that all references to a single feature flag should be contained within the domain of responsibility for single team.

#Who Can Delete a Feature Flag

Given that we have strong constraints around how feature flags get managed and referenced, it should be clear that such constraints apply to the deletion of feature flags as well. But, this is worth discussing separately because the lines of responsibility can get blurry for certain feature flag use cases.

Specifically, those that relate to A/B testing of product conversion workflows. The problem with A/B testing is that the consumption of the feature flag necessarily bleeds out of the engineering organization and into the growth and analytics organization. This creates an air of top-down control in which the engineering organization feels as though it cannot delete a feature flag until the growth and analytics organization deems it to be OK.

This is an unhealthy perspective that leads to very long-lived feature flags. Which, in turn, leads to overly complicated code that is hard to understand and maintain. Which, in turn, leads to more bugs and production outages.

This is exactly the opposite of the effect we hope to achieve with feature flags.

But, it is not the responsibility of the growth and analytics team to keep the code clean and easy to maintain—this responsibility rests entirely on the shoulders of the engineers. And, the engineers do not have to ask permission to do their job.

As such, in the case of A/B testing, the engineering organization must enforce timing constraints around the deletion of a feature flag. That is, the engineering organization must decide how long it is willing to take on the technical debt of the feature flag before the feature flag will be deleted. This then becomes the operating window in which the growth and analytics organization is allowed to conduct their experiments and identify statistically significant outcomes.

For A/B testing, this can be a negotiation between organizations. But, ultimately, the engineering team that implemented the feature flag is the only team that can delete the feature flag. And, it is the only team that can decide how long a feature flag can live.

#Security and Compliance Considerations

In some organizations, there are strict rules regarding who can deploy code. For example, some organizations mandate that the engineer who wrote the code cannot be the same person who deploys the code. In such a context, it's important that you work with the Security and Compliance teams to ensure that feature flags fall outside of this mandate.

The revolutionary power of feature flags comes from the separation of deployment from release. You must educate the security and compliance teams about this notion. If you don't, they may assume that the release of a feature is the same as the deployment of said feature; and, that any changes being made to a feature flag's configuration must be handled in the same way that a deployment is handled.

At that point, you essentially poured concrete over the workflow and destroyed the dynamic runtime value-add that is the feature flag. Do not let this happen!

Have questions? Let's discuss this chapter: https://bennadel.com/go/4552