From Laboratories to DevOps Factories for Continuous Testing

The prior blog Continuous Testing and Monitoring Essentials for SDN and NFV discussed the importance and suggested practices for testing and monitoring complex systems such as software defined networks and network functions that are virtualized . Whether the product being tested is a complex network system or not, with DevOps the laboratory is no longer just a place where engineers go to test their changes. In a DevOps environment, implemented according to best practices, the laboratory is becoming an on-line operation that is reliable, usable, remotely accessible and shareable by many types of remote users that expect 24/7/365 availability.
The old “Santa’s workshop” model of development engineering (elves) hand-crafting products in specialized local laboratories and then handing off their products to operations staff (reindeer) for infrequent delivery (on Santa’s sleigh once per year) is being replaced by a fully automated, orchestrated software DevOps laboratory infrastructure akin to a modern factory staffed by robots churning out products at a dizzy pace. Elves and reindeer need to rest between deliveries and simply can’t keep up with today’s continuous delivery expectations. Santa’s “lap-top” has become an app store where users can download products anytime no matter if they have been bad or good (but be good for goodness sake!).

If a modern DevOps factory-style laboratory is a priority on your Christmas list, here are some suggestions for tool requirements you may want to include on your letter to Santa.

Inventory tools to record and track resources:
All of the systems used in tests such as test tools, systems under test, and interconnection systems need to be tracked in an on-line inventory database in which each resource and the critical attributes are visible and addressable by administration staff and users of the resources. The ability to discover and import new or changed resources is essential. Without these capabilities it is impossible to keep up with the continuous changes that occur as the DevOps continuous test environment evolves to meet ongoing requirements.
Tools to administer users:
With a large number of users accessing lab resources features for administration of users and user groups are essential. Large scale DevOps environments require administration of hundreds or thousands of users with varying privileges for resource access according to project priorities and schedules. The ability for lab administrators to assign privileges to individual users or user-groups is essential to control access to some resources which may be restricted for security or other reasons.
Ability to create, and save test configurations:
While tests may be created to work with a variety of configurations every test assumes something about the test configuration. In a test environment implemented according to best practices the configuration of the test, or at least an abstract version of the test configuration is specified and associated with each test. Configuration data is saved in a database and may be recalled by search attribute matches when a test needs to run.
Ability to reserve tests and test configurations:
With a large number of lab users in large scale DevOps environments and limited lab resources conflict resolution features are needed. The ability assign real resources to abstract configurations provide the flexibility to use lab resources more efficiently. The ability to reserve test resources and test configurations in advance according to a calendar, or immediately when an event trigger fires are important options to minimize wasted time for both users who may have limited time windows or automated processes that have non-deterministic schedules.
Ability to orchestrate many automated tests in parallel:
In large scale DevOps environments the number of tests that need to run in a given short DevOps cycle may vary from a few tests to thousands depending on the complexity and nature of the change that are integrated in the cycle. The tools need to be able to scale the tests and run on as many physical and virtual test resources as are available, to minimize the total time for test.
Test Environment Metrics:
The entire DevOps pipeline depends on reliable, scalable and maintainable continuous testing. Built-in metrics are essential to monitor the environment’s reliability, scalability and maintainability.

The above is a partial list of suggestions for a state-of-the art DevOps continuous testing laboratory “factories” list. At Spirent we think continuous testing laboratory “factories” are critical for successful DevOps.