Feature Flags Book: Life Without Automated Testing
People who say it cannot be done should not interrupt those who are doing it.
— George Bernard Shaw (misquoted)
This may be shocking to some engineers; but, I haven't written a test in years. And, I deploy code to production anywhere between 5 and 10 times a day.
Now, I'll caveat this by saying that I don't write and distribute library code for broad consumption—I maintain a large SaaS product over which the company has full end-to-end control. If your context is different—if the distance between your code and your consumer is vast—this chapter may not apply to you.
In 2023, I published a book titled, "Feature Flags: Transform Your Product Development Workflow". This book contains everything that I've learned over the last 7 years about integrating feature flags into my product development. But, a static book can only take you so far. In an effort to make the book more interactive, I've created a series of blog posts—one per chapter—that provide a place in which the readers and I can discuss the content. You can purchase the book and / or read a preview of each chapter on the book's mini-site. Feel free to leave a question or a comment down below.
- Of Outages And Incidents
- The Status Quo
- Feature Flags, An Introduction
- Key Terms And Concepts
- Going Deep On Feature Flag Targeting
- The User Experience (UX) Of Feature Flag Targeting
- Types Of Feature Flags
- Life-Cycle Of A Feature Flag
- Use Cases
- Server-Side vs. Client-Side
- Bridging The Sophistication Gap
- Life Without Automated Testing
- Ownership Boundaries
- The Hidden Cost Of Feature Flags
- Not Everything Can Be Feature Flagged
- Build vs. Buy
- Track Actions, Not Feature Flag State
- Logs, Metrics, And Feature Flags
- Transforming Your Company Culture
- People Like Us Do Things Like This
- Building Inclusive Products
- An Opinionated Guide To Pull Requests (PRs)
- Removing The Cost Of Context Switching
- Measuring Team Productivity
- Increasing Agility With Dynamic Code
- Product Release vs. Marketing Release
- Getting From No To Yes
- What If I Can Only Deploy Every 2 Weeks?
- I Eat, I Sleep, I Feature Flag
Reader Comments
Ben,
This chapter is just wild.
However, I realized that Facebook is (was?) well known for not using QA and not writing any unit tests. They mitigated risk by dogfooding via the internal Facebook for work platform and insane metrics that came out of lower environments and alerted every baseline change. I guess that's basically what you are suggesting here but in feature-flag terms. Dogfooding by having rollout strategies where internal traffic gets features first and metrics come out of the feature management system. You are getting the big kid toys (many environments) at a fraction of the price. Let me know if you know how to get that fancy AI bug fixer program (https://blog.bytebytego.com/p/automated-bug-fixing-at-facebook) via feature flags.
In a sense, there are several strategies for mitigating risk and maybe companies shouldn't try to do all of them. Even if you have the money to burn, can you really afford the opportunity cost?
My mind still can't rationalize that you as a developer could be responsible for releasing a change without any sort of test automation or QA team. The reason I say that is that for the first part of my career, that's how we worked and those production incidents match word for word your status quo playbook. However, here you are with a couple billion dollar company without any QA team or unit tests and your main risk mitigation strategy is feature flags.
@Justin,
To be fair, the company that you reference is going to be closed at the end of 2024 😭 Though, this unfortunate end has nothing to do with testing and everything to do with a long line of unfortunate choices. It is what it is—I try not to think too much about what could have been (other than important lessons learned).
So, nothing happens in a vacuum. It's not just feature flags. It's feature flags and the mindset shift and the fact that we were mostly working in a monolithic codebase and we had a lot of people who had been working here a non-trivial amount of time and management was mostly hands-off about how we set about fixing our uptime problem (well, once we got past the era of the CCB, Change Control Board).
To some degree, it was kind of great that things were so terrible before we introduced features flags—it meant there were very few expectations about how it had to work. Which gave us a lot of freedom to experiment. This is especially true in regard to tests - we had no rules about testing (such as test coverage goals).
And, to be clear, we did made a lot of mistakes. Even today, if you look in our LaunchDarkly dashboard, we have 1,370 feature flags. Which is pure madness! A lot of teams got really sloppy about cleaning up after themselves.
But, this huge number of flags is also an indication of how heavily we used feature flags. They allowed us to start working really small and focused. And once we leaned hard into that level of focus, we had to start architecting the code in a way that inherently reduced the numbers of bugs.
In a real way, I see features flags has having the same influence as tests. Meaning, when you write tests, you obviously get the assurances of the tests themselves; but, in addition to that, people usually have to start thinking about the way they write the code in order to make the code easier to test. As such, they start to create code that is more isolated, more focused, and depends more on Inversion of Control (IoC). This, in turns, tends to make code that is easier to reason about and easier to maintain. So, not only do you get the test artifacts, you also tend to get better code.
Feature flags, at least for us, did the same thing (altered the way we went about writing code such that it inherently became easier to read and maintain).
Of course, everyone's mileage may vary. And, if you're on a team that mandates test coverage or has a very strict QA process or only deploys code on a periodic basis, you definitely lose some of that ability to work small and lean on feature flags. Not to say they no longer help—only that they won't bring all the same benefits.
Thank you for engaging with the book! I'm super happy to answer any questions or challenges to the content. At the end of the day, it's only one company's experience.
Post A Comment — ❤️ I'd Love To Hear From You! ❤️
Post a Comment →