Feature Flags at Building Radar
If it hurts, do it more often
In the summer of the past year, I joined Building Radar as a contractor developer. The team recently launched a new product, onboarded some clients, and was looking for ways to move forward faster.
When I joined, the team was releasing a new product version every three weeks, and the release process was somewhat painful. Things fell apart when deployed to production, migrations took longer than anticipated, features didn’t work quite as advertised, etc.
The solution to this problem is to release more frequently, even though this may seem counterintuitive. Martin Fowler calls it, if it hurts do it more often. In a team discussion, two arguments were raised against more frequent releases:
- We don’t want to ship incomplete features.
- We don’t ship features before announcing them and training the customer support team.
To address these concerns, I suggested adopting release feature flags. The power of feature flags is decoupling deploys from releases. In software development, “deploying a new version” and “releasing a new version” are often used interchangeably. Before feature flags, I never considered separating these concepts, but the whole thing clicked once I embraced the idea.
Deploy is a technical term when developers roll out a new codebase. New code may contain new features. Still, they are are hidden behind feature flags (in simple words, disabled by a configuration option.) For an external observer, the system behavior is the same.
Release is a product term. When we release a new feature, we turn on a related feature flag and make the component visible and usable to clients.
Decoupling these two concerns, you let the development and the product team work more independently. Developers and product managers don’t need to agree on when they release what.
Feature flags with DevCycle
At Building Radar, we ran a spike to experiment with multiple solutions. We considered Open Source solutions that we could host ourselves, and a few cloud solutions. In the end, we chose DevCycle, which offered all the features we needed, libraries for all the platforms we required, and an affordable pricing model with a generous free tier. Although I found the dashboard cluttered and not intuitive, we decided that it was a minor issue.
Fast-forward a few sprints, the team was making several deploys per week, and the process became less stressful and more predictable. Our ultimate goal is to move towards on-demand deploys triggered by developers.
Meanwhile, while working with feature flags, we learned a few lessons that marketing materials of their service don’t emphasize.
Feature flags lifecycle
☝️ Have a process for deleting feature flags once the feature has been released
One of the potential drawbacks of using feature flags is that they can cause confusion and clutter in your codebase if not managed properly. Essentially, each feature flag is an if-else clause. Once a feature is deployed to all users, the “else” branch is no longer executed, but the dead code remains and can slow down your code.
At Building Radar, we have a rule to delete feature flags as soon as the related feature is released. During each sprint, we create a Jira task to remove the feature flags and any leftover code from previous releases, and this task is treated with equal priority as other tasks.
I particularly appreciate working with Building Radar because they are proactive in establishing processes. Whenever they identify a recurring pattern, they set up a process and the team lead ensures it is followed.
Don’t be afraid of processes; they are beneficial for repeatable actions. If you are not convinced, I recommend reading The Null Process by Kate Heddleston.
And by the way, if you feel like working with a team that is dedicated to processes, check out Building Radar’s careers page. I recommend them because I like working with them.
What to hide behind flags
Not a pitfall but rather a remark. It’s important to note that in a typical frontend and backend web application, release feature flags are most often used on the frontend side.
Indeed, when releasing a new feature, backend developers usually expose new API endpoints or extend existing ones with extra parameters. If the changes are backward compatible, no backend feature flags are necessary. The feature can be hidden in the UI.
Feature flags beyond releases?
Feature flag apologists argue that you can use them for everything: from gradual releases to A/B testing to kill switches, to implementing tariff plans, etc.
Anecdotally, I heard stories of hard-to-maintain messy systems that used ad-hoc feature flags as a replacement for their tariff plans. There, they enabled and disabled tons of features for different clients. The uncontrollable proliferation of flags resulted in implicit and hard-to-debug dependencies between flags, where certain flags work only in a specific combination.
At Building Radar, the team chose to use feature flags only for releases, which helps keep the flags’ lifespan and scope limited. This approach allows them to get more familiar with flags and gradually introduce flags for other purposes if needed.
By the way, if your concern is tariff plan implementation, read the post you should separate your billing from entitlements.
Dependency from an external service
Your system’s reliability is only as strong as its weakest link, and you don’t want your feature flag solution to be that weak point. We learned this lesson the hard way in our first month of using the solution.
The day after we released and announced new features, they suddenly disappeared. At first, we suspected someone had accidentally turned off the flags, hiding the new functionality. However, we soon discovered that the DevCycle team had made an unfortunate update to their service, which set all of our feature flags to False.
The DevCycle team fixed the issue even faster than they introduced it in the first place, but this experience served as a reminder that relying on third-party services can lead to failure if those services go down.
For those who are curious, the fix they made to their Python SDK can be seen in a Pull Request #11.
To minimize the impact of similar incidents in the future, dedicate time to regularly removing unused feature flags.
Performance impact
It happened to me before, and it happened to us this time again. A deeply buried feature flag that helped resolve object features got resolved hundreds of times in a loop.
This type of loop can happen accidentally, for example, when you check the value of a feature flag in a GraphQL resolver and call that resolver for each object in the returned result.
When we discovered this problem, we implemented feature flag caching. While the DevCycle SDK didn’t include feature flag caching by default, implementing it was straightforward.
Different environments for development and production
☝️ Don’t forget about the OFF state.
It’s common for the feature flag to be always ON in development and staging environments, but when you release it, it is OFF by default in production. This can result in a whole set of execution paths going untested in production.
We’ve encountered this issue more than once as we’ve become more familiar with feature flags. A solution is to thoroughly test both flag values.
- Development: When developing a feature, toggle the flag on and off to test its behavior.
- Unit tests: If necessary, create tests for both values of the toggle. Python and Pytest’s parametrized fixtures can help you run tests for both flag variants.
Takeaways
Feature flags are a pretty cool way to streamline the dev process and separate deployments from releases.
But, like with most tech solutions, there are some trade-offs to keep in mind. You’ll have to deal with dependencies on external services, extra testing challenges, and potential performance issues. It might take some time to adjust and get used to, but if your team is up for a challenge and open to trying new things, it’s totally worth it.
If your team is feeling bogged down by communication and coordination problems during development, give feature flags a shot. It might just be what you need to speed things up.