monday.com Engineering: Why We Develop in Production
Our developers here at Monday aren’t just executors of code; they’re brilliant end-to-end thinkers who are incredibly independent and self-managed. They take full responsibility of the features they build from conception to post-deployment, and are all super nice guys to boot.
The work they’re doing on our product is pretty cutting-edge. If you’re a developer or tech enthusiast, we thought you might be interested in learning a bit more about what they do, so we’re happy to kick off a new Engineering series. In this post, Fullstack Developer Aviram Gabay shares about why we develop in production. Enjoy!
3 things this guy would take with him to a desert island: His guitar, his mom’s cooking, and clean pajamas (island or not, he’s not going to sleep in dirty clothes)
TL;DR: Building your features in small increments and deploying each to production has a lot of benefits. It makes it easier to spot bugs and deploy, and it lowers stress the day of the release. This method cannot be applied for all cases of feature development, but it should always be considered as a valid option.
New project: monday in different currencies
Over the past few weeks, I’ve been really busy working on a new project. Before, we only allowed our customers to pay in US dollars, and we’re now displaying and allowing people to pay in their local currencies.
This project has involved modifying many different areas in our platform and codebase — from allowing transactions in multiple currencies and displaying the correct currency for each account to adjusting our invoices to support multiple currencies, to updating our homepage…and much more.
Each of these sections is highly sensitive. If I break something, it means either our clients can’t pay or they will be displayed/charged with the wrong amount.
Sounds pretty stressful right? Well, by using development in production we were able to eliminate much of the stress.
So what does it mean to develop in production?
It means breaking down your project into many small, non-dependent tasks. You can then deploy each mini-task to production once it’s completed.
Breaking the project into small non dependant tasks is not always trivial. We aspire for each small task to behave as a small product increment, meaning it will not break the product even if it ran, but it does not mean it contains the completed set of capabilities of the complete feature.
So what prevents the code from running? We use Feature Flags, a system of switches that we can turn on and off whenever we want, to assure this code will not run until we want it to.
This way the new snippet of yet unused code resides with the existing production code while it is still under development. When the new feature is ready we simply need to flip our feature flag to turn it on to use it.
For example, with multiple currencies, the project was built mostly around a new single method (or more accurately a service) which was my source of truth regarding the currency the user will see. Using a feature flag, I was able to force it to show USD until the feature was complete.
Having this allowed me to break down the task into a few small, non-dependant product increments, such as:
- Create a new endpoint in our API that fetches our pricing plans in the correct currency. This endpoint was used by other team members from very early stages of development.
- Add the ability to make a transaction in non-USD currency. This feature was active once it was deployed (but the feature flag made sure it would continue to do USD transactions)
- Update all the invoice/promotion templates to support multiple currency types.
- Update our admin area (which shows a mini version of our invoices and current plan)
- Add the supported currencies to our plans
- Add support in our website to different currencies and make sure users will see the same currency in the website and the app
Hold your horses. Isn’t it dangerous to deploy non-fully functional code?
Not at all! It’s actually much safer to deploy snippets than a huge chunk of completed code. Even when every line of code is fully tested, bugs/regressions are bound to happen. When they appear, it’s a lot easier to spot the problem between 50 lines of code than 500.
But wait, there is more! More benefits of working in production
Deploying small tasks also makes code reviews much smaller, thus, increasing their quality and speed.
Deploys are faster as each task branch stays relatively aligned with the production branch which significantly lowers the possibility for major code conflicts. It also makes it easier for members of your team to sync with your project as it is already in production.
If you make any changes in the database schema, breaking them into a lot of small changes makes it easier to prevent corruptions, do rollbacks if needed and with as little downtime as possible.
One of the biggest advantages of this method is that by having the old and new code together, you can AB-test the effects of your changes on your clients. Using the Feature Flags system I mentioned before, we can open the feature/update to a group of clients and compare with another group that doesn’t have it. We do this with all of the features we release to make sure we are always improving and never harming our clients experience.
Bonus Points: Early feedback & empowering our clients
Being able to turn a feature on and off allows us to grant it to ourselves in early stages of development so everyone in the company can play with it. Moreover, we can grant it to specific client accounts as a beta feature. This allows us to get feedback and make adjustments much sooner and easier than if the feature were already finalized. Also, this empowers these clients, as they have exclusive access to features that are still not available for everyone. It’s basically a win-win for everyone.
Impressive! But it can’t all be good, right?
Well no, not everything is perfect. For example, different reviews for each task might cause them to be out of context. Adding unused snippets of code might look confusing for someone who is not aware of your project. Usually there is some development overhead that is required to make sure both the current code and the new code can live together. And once the new code is up and running, some cleaning up is usually required to remove all the traces of the old code.
That being said, most of these issues are easily solvable. You just assign someone to review and monitor the whole process. It’s also important to maintain proper comments in the code for other members in the dev team and to make it easier to clean up.
Thanks, Aviram! You’re the best. Stay tuned for our next engineering post!