Animoto’s web and mobile platforms enable businesses and consumers to drag and drop stock and user-defined visual and audio elements to create videos via a tiered monthly subscription service. Animoto updates service features with increasing frequency—some of them several times a day.

“We have a broad range of applications and features including an online video maker application, where we enable users to import videos and video elements from all the known, established sources such as Facebook and through Dropbox,” says Siddharth (“Sid”) Jain, VP of Engineering at Animoto. “They can then export videos out to a variety of venues such as YouTube.”

Animoto must iterate new innovations continuously into its application/platform for high-quality yet simple video creation to compete with similar solutions in the marketplace such as WeVideo and Stupeflix. This fast-paced development process is challenging, since Animoto must update separate versions of its software for the web, Android OS and Apple iOS (both iPhone and iPad). “The mobile apps are largely consumer-focused for very easy, on-the-go, frictionless video creation and sharing, while our web apps feature server audiences of professional photographers, educators and small businesses,” says Jain.

In addition to testing service capabilities of different versions of the video maker application, Animoto must test the on-boarding/registration flow and the free product experience, where users get their first taste of Animoto. “We do lots of interesting things such as testing support for multiple currencies around the globe and running continuous multiple AB tests to ensure that the free product experience is working,” Jain notes.

Notable Animoto testing includes user funnel testing in a simulated user flow environment starting at sign-up and continuing through video creation, feature exploration and the upgrade experience. “We test this several times a day on the different platforms,” Jain says, noting Animoto tests for the seamless success of import and export of videos and video elements to and from Facebook, Dropbox and YouTube. “The emphasis is on ensuring the quality and consistency of the user experience, i.e., making sure that nothing goes out that breaks revenue in terms of service.”

Porting to Sauce Labs Promotes Progress

Sauce Labs‘ cloud service helps Animoto scale up and out instantly to test Apple/iOS versions of its application so it can meet the high standards set by Apple and ensure fast inclusion of new app versions in the Apple App Store. This helps Animoto to compete by getting new innovations and features to market more quickly. “The biggest advantage to using Sauce Labs is that we don’t have to create or maintain a test lab internally,” says Jain.

This accelerated testing speed is especially important since the app must adapt to every new iOS version while maintaining backward compatibility. “It would be a very nightmarish thing to do this all manually,” he says.

In addition to faster testing, Sauce Labs is helping Animoto ensure that testing was delivering on its promises of software improvement through accurate feedback. “We used to be religious about doing extensive unit tests. Now we emphasize integration tests over unit tests because we got bit by unit tests that were all green even though the software functionality was broken,” says Jain. “Now we have a balance between unit and integration tests while frontloading the latter with tests of customer-facing functionality and collecting fine-grain feature use data.”

The comprehensive testing helps to ensure that users aren’t shying away from certain features and that features are not broken. Frontloading uncovers regressions very early so the Animoto developers dedicated to fixing them can catch them before they lead to breakage.

Sauce Labs helps Animoto shrink technical debt. “Debt rears its head in awkward ways. We do get surprised, but having good test coverage allows us to know where the app hot spots are and to have feedback to fix these and verify changes swiftly,” says Jain.

Results: More Stability, More Capacity

“We have been able to reduce testing time from three hours to under 30 minutes for some tests,” says Jain. “Thanks to Sauce Labs’ cloud, we added VM capacity, moving from 20 VMs to 45 VMs for the testing environment.”