By Mohit Goenka, Senior Engineering Manager & Nikita Varma, Senior Software Apps Engineer
When you’re building an email inbox for millions of users, you need to make sure its deployment is smooth, fast, and scalable, as well as capable of performing tests across environments to ensure a quality experience.
The Yahoo Mail Android app is Verizon Media’s most popular Android mobile app, so as the engineering team prepared for this major update, we prioritized developer productivity and identified 5 key principles to manage the delivery of the new Yahoo Mail experience:
As a standard practice, the Yahoo Mail Android Team pushes new releases to production on a bi-weekly basis, but we also adhere to the continuous delivery principle, which encourages more frequent updates. Because our build pipeline is CI/CD compliant and fully automated, we push out changes, such as new features, bug fixes, configuration updates, etc. within just a few hours every day. This helps us deploy adjustments quickly, ensuring our users receive the best Mail experience possible.
To make sure features and changes are ready for our consumers, we follow a strategy of staged rollout and test various environments. The code changes are tested thoroughly before roll-out in three stages: alpha, beta, and production. Alpha is a closed internal channel for Verizon Media employees, where they are able to give direct feedback/report issues. If there are no issues in the alpha channel, the same Android Package (APK) is promoted to the beta channel: open channel with external users for their feedback. These two channels give the team the ability to analyze quality metrics before rolling out the build to users in production. To make sure the code that reaches production users is bug-free, instead of directly releasing the changes to all the users, it’s rolled out in phases. First, it’s rolled out to a small subset of production users, and if all the key metrics and stats look stable, then it’s pushed to a wider audience. After it is thoroughly baked with a wider audience, then it is released to all the users. All the while, we continue to monitor for user feedback, bugs or issues - so we can react fast in resolving them.
This complete process ensures that our code changes are fully verified before hitting all our production traffic.
Staged rollout and support for these various test environments typically require the teams to build, maintain, and support various configuration files for each test environment and build and upload multiple APKs for each of these. This is not a scalable approach and requires a lot of manual effort. To solve this problem, the Android team built and uses a single unified APK that can be pushed from alpha to beta to production channel without having to build, generate, publish and maintain multiple APKs.
The team has also automated the deployment of an APK directly to the Google Play Store using the Triple-T Gradle Play Publisher plugin. The team has configured various build steps and done customizations on top of the plugin to achieve the desired result of publishing the APK at a certain time interval during the day. This process is much faster, requires no manual intervention and the team doesn’t have to worry about uploading the APK to the Google Play Store alpha channel manually.
Automated scripts are built in the pipeline that notifies developers of changes during various stages of deployment. For example, once an alpha build goes through, the developers who contributed to any release through code commits or by merging pull requests are sent an email notification. This helps the developers, QA team and product managers to test out their changes faster in various environments.
Even with thorough testing, a bug makes it to production once in a while. And we detect these via closely monitoring and interacting with users around their feedback or issue reports. For such instances, we have code patches called hotfixes that we quickly deploy on top of production code to address critical issues impacting large sets of users. This way we ensure that at any given point in time, the app that is listed on the Google Play Store is stable and bug-free.
As part of our continuous delivery efforts, we have also developed a concept we call kill switch and nag. Users are notified of important builds that they should immediately update to, which can include bug fixes, product updates, or new features. The kill switch prompts the users to update the app and doesn’t let users use the current version, while the nag feature lets the user choose to either continue using the current version or update to the latest one. This allows us to notify the users about the latest version of Yahoo Mail when we see that a user’s current version needs to be updated.
In building the new Yahoo Mail deployment strategy, we knew that we needed to revamp from the ground up, starting with our continuous integration and delivery pipeline. The guiding principles of our new, forward-thinking infrastructure allow us to deliver new features and code fixes at a very high launch velocity and ensure that our users are always getting the latest and greatest Yahoo Mail mobile applications. The process is not just simpler and agile and saves developers a lot of time, but it also is faster, error-free, and completely automated. The Mail Android team has built a generic solution for the automation of deployment of an APK to the Google Play Store that can be incorporated in any Android app at Verizon or Verizon Media. The standard the team has set and followed is now serving as a standard CI/CD definition for all Android apps across Verizon Media.
Head to Google Play and download today - we’d love to hear what you think!