Mobile developers understand the rising importance of continuous integration (CI). The promise of CI is to speed up the overall velocity of the app team by building and testing every single code change. This environment of rapid change means that code integration issues are caught early, and testers and beta users can provide feedback on these changes as quickly as possible. As CI has become commonplace for app development teams, and many are moving their CI infrastructure to the cloud, the importance of build and regression test automation has grown.
To that end, we thought it would be important to generate a research report by examining survey data from more than 500 mobile app developers and benchmarked usage data from more than 250,000 automated builds, tests and releases on the Ship.io platform. This Mobile Metrics Benchmark Report shares insights and trends around the Continuous Integration (CI) landscape in regards to mobile app development. Key areas of focus for the report include CI adoption across major mobile operating systems (iOS, Android), frequency of testing, releasing and checking code, app build times and frequency, and common build failure reasons across geographies around the world.
OVERALL CI ADOPTION
Of app teams surveyed, the majority are leveraging CI. Those not leveraging CI are often building only on-demand when they want to test a build, or are building with a simple regularly scheduled nightly or weekly build scheduler. Of teams that are leveraging CI processes, the majority of them are running some form of test automation on each build. Here’s a breakout:
- Less than 5 per week: 15%
- Between 5 and 25 per week: 33%
- Between 25-50 per week: 33%
- More than 50 per week: 19%
- No, not doing CI right now: 37%
- Yes, doing CI builds and tests: 35%
- Yes, doing CI builds: 28%
THE CLOUD CI ADOPTION LANDSCAPE: Cloud CI Adoption: iOS vs Android
Many app teams have adopted Continuous Integration as part of their engineering processes. Some teams have set up CI as an on-premise service, but many are moving this to the cloud for reasons such as cost savings, focus, and reliability of the CI system. An oft-cited reason for moving to cloud CI systems managed by others is the complexity and cost of keeping up with all the changes in the iOS and Android build and test ecosystem.
Of all developers surveyed with an interest in Cloud CI, but not necessarily using CI systems today, the majority (50%) are only focused on iOS—ignoring Android altogether. Other developers are focused on both platforms, with a small minority focusing only on Android.
PUBLIC vs. PRIVATE APPS
Developers with an interest in CI are overwhelmingly building apps for Google Play and the App Store. Only a small fraction of these developers are building apps for internal use, or with a limited number of customers. Many of these internal-facing app teams have smaller app teams and longer release cycles, presumably diminishing their need and interest in continuous integration systems.
WHY BUILDS BREAK AND BUILD SPEED METRICS
As many app teams have moved to a continuous integration (CI) model for development, they produce many more builds. This increase in build frequency means that the time spent building is far more significant, and broken builds can randomize and delay product progress. The data shows that the majority of build breaks are avoidable, and the time spent building apps is low, but iOS builds are about twice as fast on average versus Android.
Analyzing the source of build breaks across both iOS and Android reveals that there is a very long tail of build breaks as many engineers would suspect. Builds often break due to of various developer code syntax issues, but several build breaks account for the vast majority (74%) of all breaks in cloud CI builds:
- Missing File or Directory: Developers often forget to check dependencies and new files into the code repository.
- Lint Errors: When ‘lint’ is enabled on a build, it often catches common programming errors.
- Unit Tests: Many apps have regression test automation at the unit level. These tests fail during the build process, catching either changes in the app, or regressions.
- Linking: Builds often fail at link time because a referenced 3rd party library wasn’t added to the dependencies list, or project dependencies were not updated.
- Code Signing: The process of signing apps is still error prone. From missing signature files, or differences in debug versus release build configurations, code signing causes many CI headaches.
The good news is that most all of these build breaks could be caught by developers having a separate ‘clean’ enlistment and building and running lint locally. Even more breaks would likely be prevented if developers simply executed unit tests before submitting new code. Barring that, cloud CI systems make it easy to catch all these issues, and mark the build as failed until developers resolve the break.
BUILD TIMES: iOS vs. ANDROID
When app builds break the team is often blocked and waiting for a new build. Build times are now a critical path for app teams. Analysis of iOS versus Android build times reveals that although iOS app build times are about twice as fast on average, the build times are still reasonably short at 2 minutes and 4 minutes on average, respectively. A deeper look into the build time data shows that very few app build times, regardless of platform, exceed 30 minutes. The majority of app build times are less than one minute. An interesting aspect of build times is the fact that more Android than iOS app builds complete in less than 2 minutes. But of all large/complex app builds (longer than 2 minutes), iOS app build times are generally faster by a factor of 2.
COMMIT FREQUENCY OF IOS vs. ANDROID CLOUD CI TEAMS
Of the top 40 apps with the most frequent commits in the study data, 67% of them were iOS Apps. It is worth noting that the study data had builds from app teams that built both Android and iOS versions of their apps. iOS developers seem to be committing code more frequently and adopting the CI model of small, continuous commits.
COMMIT FREQUENCIES: IOS vs. ANDROID
iOS developers make far more revisions than Android developers. Across the spectrum of commit frequency, iOS developers, on average, are making twice as many commits as Android developers. The majority of apps have one commit per week, or fewer. Of those that are very active, defined as committing code more than 3 times per week, about two thirds of those developers are working on iOS. The aberration seen where Android developers are building 6 times a week, is due to a single app vendor with many apps, all sharing a common code base with many Android apps (with commit frequencies at 6 times per week). In general, iOS developers are about twice as likely to make more than 3 commits per week versus their Android counterparts. In total, iOS developers, on average, are making twice as many commits as Android developers.
TOP PRIORITIES INCLUDE FASTER RELEASES
Of all the things app teams were looking to prioritize in the coming year, faster release and update frequencies (41%) was a close second only to improving overall app performance at 48%. Teams are quickly migrating to Cloud CI systems as a way to accelerate their overall velocity. Cloud CI enables teams to focus their time and money on other priorities such as performance and security.
WHY NOT CI?
The reasons why teams haven’t adopted CI vary widely. 49% are interested and plan to move to CI eventually but haven’t because of prioritization. Some teams believe they are too small to benefit from the value of CI as their team only has one or two people working on the builds, or they have so few code revisions that they prefer to deal with builds manually. A minority of app teams are concerned with the cost in money or time to move to a CI system.
These Ship.io study results show that the world of CI for app teams is changing quickly and varies between iOS and Android app teams. Knowing the macro-level trends and learning from them, can greatly impact your app teams efficiency. As app teams accelerate due to CI adoption and agile practices, teams are often spending an inordinate amount of time focusing on their local and custom build and test infrastructure. Those app teams that have moved to the cloud are saving money and time and focusing more on their app design and implementation.
- CI Adoption: Cloud CI adoption is on the rise, especially in the iOS community. iOS developers are more aggressively adopting cloud CI than Android developers. The number of teams adopting Cloud CI is doubling every 6 months.
- Commit Frequency: There is a wide range of commit frequencies across iOS and Android. iOS app developers generally have twice as many commits as Android developers. The commit frequency and the number of builds range from a high of around 16 per day, to a low of once every other week.
- Source Control Popularity: Two cloud source control repositories represent almost all cloud CI systems: GitHub and Bitbucket. The momentum is strong with Bitbucket which saw its share of usage double in just the past 4 months. Android users seem are more aggressively moving to Bitbucket, but iOS developers aren’t far behind.
- Build Breaks: Several common issues account for the majority of build breaks, but there is a very long tail of build break reasons. The good news is that many of these are avoidable.
- Build Time: Build times on iOS and Android are similar and range from mere seconds to a high of 40+ minutes for apps with larger code bases and dependencies.
About the Author/Prathap Dendi
Prathap is GM for Ship.io, and VP Business Development for Electric Cloud. He drives strategic partnerships that help us deliver broader solutions to our customers. Electric Cloud has forged strategic partnerships with companies providing industry-leading tools and services in order to deliver software production automation to our customers.