Advantages of software update automation in high-end networks

Software in today’s high-end networks

Modern-day high-end networks are based on two key elements: specialised hardware for packet forwarding on the data plane and specialised software on the control plane.

The software on the control plane serves as the system hub. From here, the data plane is programmed to exploit the full potential of all network components – by using various protocols and routines, which in a high-end routing system can quickly add up to hundreds of processes at the operating-system level and more than several thousand tables.

In addition, the control plane serves as the primary repository for exchanging information with the systems, because the components are in constant exchange in the live network, for example by means of:

other network devices via routing protocols such as BGP, ISIS, OSPF, LDP, RSVP etc.
orchestration systems via protocols such as NETCONF, REST APIs etc.
SDN controllers via BGP-LS, PCEP etc.
the operator via SSH

These protocols are only a small overview of what can be found on a network operating system (NOS) today.

The software on the control plane plays an elementary role in network operation, keeping it up-to-date is crucial: It is the only way, for example, to eliminate bugs and critical security flaws, or to map new protocols and functions, support new hardware and ensure reliability.

However, as operators of high-performance networks will be all too aware, introducing a new software package into a network is not a trivial matter. Before implementation, careful checks are required to determine whether the new software fixes known problems, correctly implements new functions and does not introduce any new bugs to functions that are already in use.

The challenge of software updates

To minimise the risk of problems in network operation caused by the introduction of new software, many of our customers run intensive software release tests. Depending on the complexity of the component(s) to be tested, this can take weeks or even months, meaning that the introduction cycles of new software often become increasingly lengthy. This can turn into a vicious circle, due to the long intervals between the test cycles, the defined test set-up and the live network diverge more and more – which in turn means that the testing engineers must first re-familiarise themselves with the test scenario and potentially extend it to include new test cases, making each individual test run even more time-consuming.

In addition to running the test itself, planning and documenting it requires enormous efforts: Typically, documents are created to describe and define the tests, which are then carried out accordingly, any deviations being documented manually. With complex tests in particular, this often results in documents spanning several hundred pages – which are of course not created purely as an end in itself, but which for many network operators also serve as important proof of compliance.

All in all, the process therefore requires not only a suitable lab environment, but also considerable time and know-how on the staff side. Perhaps unsurprisingly, as a service provider in the network environment, at Xantaro, we have in recent years noticed an increasing customer demand for our testing services. That simple fact, however, did not magically eliminate the challenges of qualifying new software releases!

A new approach to software testing: from an idea to a solid solution

Having taken charge of test projects for our customers, our engineers ended up being repeatedly tied up for long periods of time. In addition, we were not satisfied with the delay between the selection of a software release and the provision of the lab test results. A new approach was needed: one that would allow us to quickly and accurately run a pre-defined test plan.

It was clear that the solution could only lie in automation and, in our experience it would have to meet a series of requirements for automated testing:

vendor-neutral or cross-vendor
open for the integration of external test tools such as Ixia IxNetwork, TREX etc.
definition of test cases in text files to enable versioning with git
based on existing OpenSource tools wherever possible
simple operation without in-depth programming knowledge
automated implementation of test routines, configuration changes and software updates
test results automatically processed into PDF reports

And so, after defining the framework and over the course of several months of development work, we came up with a brand-new tool that is now widely used in our XT³Lab as well as in our customers’ labs alike: XFAST – Xantaro Fully Automated Software Testing.

XFAST – practice shows the advantages of software update automation

The ever-growing number and varieties of test scenarios increasingly also reveal the potential of the solution:

Effort – a few hours instead of several days

For instance, in the context of our NextGen Data Center Solutions, where XFAST is continuously used to select stable and functional software, we were able to cut the labour of five working days of manual software testing down to a mere four hours.

Preciseness – simply more details

And the advantages in terms of accuracy, too, are hard to overlook: Previously, the scope of testing was limited by the amount of time required, today we can test much more and thus in greater detail – because it doesn’t make much difference whether a test run is launched shortly before the end of a working day and runs overnight or at the weekend, taking four, eight or even 24 hours.

Test Options – also long-term tests are possible

This means that, in addition to functional tests that give a solid first impression of a new software release, it is also easier to run long-term tests that constantly trigger events in the test topology and thus keep the components that are being tested busy. Against this backdrop, to use a concrete example, last year we expanded a software release test for a customer and included a long-term test component: In an XFAST test suite, we defined several different convergence events in the test topology, which had a total runtime of only a few minutes, but was initiated every 15 minutes over the course of four days.

Results – automated reports, reproducible for new releases

At the end, the XFAST test suite reported over 300 successful test runs, but also problems in a single-digit number of test runs. An analysis of the XFAST reports then identified a ticket at the vendor end, who, based on the information we provided, was able to recreate the problem cases and solve them in a service release, which then ended up producing a clean test run in all cases.

These and other practical examples highlight the advantages of automation over and over again: Essentially, it doesn’t matter whether we’re talking about implementing new features or systems and their interoperability or about bug fixes and eliminating security flaws – automating your software testing processes alone already saves an enormous amount of time and cost, and it increases security, especially in the continuous development of ever more complex network scenarios.