Software Testing Outsourcing

Traditionally, software testing has been an in-house function or a task performed by software vendors as part of a complete development/testing/implementation package. However, as IT has become more critical to business performance, the role of testing has become increasingly important and even strategic. Today’s businesses rely on cutting-edge technology that is rolled out quickly and performs reliably time after time for the end-user. A single application failure at a crucial point in a process or transaction can be both expensive and complex to repair, as well as costing businesses millions in lost sales and customer goodwill. When a single high-profile outage could irreparably damage a business, it’s vital that upfront testing is as good as it can possibly be. So there is a growing awareness that good software testing is a specialized and professional skill, and not merely an afterthought slotted in at the end of the IT project lifecycle. But meeting this challenge can be difficult for IT leaders who are facing a scarcity of skilled software testers, and limited testing resources. On the face of it, outsourcing should solve a number of issues including quality of the ‘go live’ software and limited skills resources. So how can we account for the reluctance or indeed suspicion of many enterprises to entrust their software to an outside supplier, particularly if taken off-site and even out of the country? Generally, software testing is still a relatively immature function within the IT department, and this can make it difficult for CIOs to clearly understand how it might best be outsourced. A common misconception is that outsourcing a problem just results in a bigger, more expensive problem. Relatively few organizations today have clear, consistent standardized testing processes, particularly in distributed multi-site organizations, so the concept of passing responsibility for fragmented testing processes across different lines of business can seem risky. Another issue is that many companies do not have a standard baseline or internal benchmark for the cost of existing in-house testing, and so have little clarity of the extent of potential cost savings, or reassurance that the stated savings can actually be achieved. There is also limited appreciation of the depth of specialized skills involved intesting customized and packaged applications in a variety of IT environments and the benefits of separating development and testing. Many companies believe that testing can only be managed in-house because of the need to understand the intricacies of the business and its processes. And, moreover, the belief that an enterprise’s own test processes are not well defined or standardized can suggest that outsourcing is difficult and problematic. So the key barriers to successfully outsourcing software testing are certainly varied but, they can be overcome. Selecting an experienced testing partner with proven industry expertise and structured processes, which will ensure best practices are consistently applied, is a proven way forward for many clients considering outsourcing software testing.

A Structured Approach to Outsourcing Testing Can Achieve Cost, Operational and Performance Benefits

Outsourcing application testing can deliver a range of benefits and also help to embed quality as a strategic competency in customers’ businesses. Realizing cost savings of 50% or more can be achieved through:

• Use of a specialist independent testing supplier;
• A clear financial and business case as an essential baseline;
• A business-driven structured testing methodology;
• And the discipline of performance measures to ensure providers really deliver.

This approach, which Forrester describes as “a much more ‘activist sourcing’ approach” 3 , can resolve the problems cited and this business white paper examines the essential components that collectively provide reassurance that outsourcing can deliver testing quality and business value.

A Specialist Independent Testing Supplier Provides Reassurance

Many organizations believe that software testing can conveniently be carried out by a vendor responsible for development. On the surface, this makes good sense: the development vendor knows the product and the client, and should therefore be able to do both jobs equally well. And simplified vendor management has its apparent advantages; a single contract that specifies testing and development means one less vendor to manage, and, in theory, easier communication. However, outsourcing development and testing to a single vendor can be extremely problematic. There is an implied conflict of interest between developing and testing the same system: can you really rely on one company to fully and exhaustively test its ‘own’ software, particularly if the supplier believes that it has met all its requirements or expended all the hours allocated to build requirements? Contracting a single vendor to provide both development and testing can reduce supplier management costs, but it is unlikely to deliver the best business value.

A developer may have a interest in completing testing quickly or in finding fewer errors, and is unlikely to have the same degree of expertise as a dedicated testing provider. Using a specialist provider for testing gives IT leaders the clear reassurance of an independent auditor with no vested interest in completing testing too quickly, or in finding an encouragingly small number of defects. Of course, there may be instances where software testing and development is necessarily conducted by the same supplier because of commercial or technical limitations. However, in this scenario, it is vital that CIOs insist on separate entities for testing and development. Obviously splitting into two teams means there are clearer lines of responsibility—each team should have a separate contract and separate KPIs to ensure that overall quality is not undermined by a lack of clarity. A concern of some CIOs is the belief that a provider cannot effectively test software without expert knowledge of the client’s business and industry sector.

Certainly, every organization has cultural and sectorial aspects that shape the business and therefore the testing requirements—but this is not in itself an argument against the feasibility of outsourcing testing, or of using a specialist testing services provider. Part of the value of a successful outsourcing agreement is the ability of the service provider to bring together in-house and out-of-house skills and knowledge to deliver the best possible service. And these skills have usually been acquired through working in a wide variety of industries and organizations. It is important therefore to look for organizations with broad, demonstrable experience of working with a range of enterprise applications (SOA, SAP etc.) and industries, especially regulated sectors such as finance and pharmaceuticals. SoftwareQ’s approach to outsourcing brings together the best talent and specialist sector skills, to provide the optimal balance of cost and experience.

A Clear Financial and Business Case is an Essential Baseline

Often improving the efficiency and quality of software testing via outsourcing is an ideal opportunity to cut IT costs. With IT spending under constant pressure and many CIOs increasingly facing budget freezes or cuts, outsourcing of software testing can reduce the cost of maintaining a large in-house testing team, enabling the client team to focus on adding value elsewhere in the IT organization, often taking on a more strategic role. Key to understanding how cost savings can be achieved is the creation of a financial/business case for software testing that clearly identifies the areas for improvement. This project approach, based on a business case, ensures that the tasks achieve a predefined business result, within a defined lead- time and within budget.

Moreover, the risks are made visible and are mitigated as far as possible. This business-driven, test management approach often presents a considerable challenge as very few IT organizations have an accurate understanding of their own current testing costs, because they are not separately measured or controlled. As a rule of thumb, we estimate that total executable testing time actually accounts for between 30% and 40% of total project costs. If software quality is low, this figure can be significantly higher, because the cost of fixing production errors increases as the project progresses. Last-minute fixes—a common problem if testing is ineffective—can be extremely labor-intensive and expensive to address. So given the relatively high proportion of costs that are attributable to testing, it is likely that some efficiencies can be introduced and resulting savings made.

A Business-Driven Structured Testing Methodology Applies Rigor and Efficiency

Of course, these cost savings are not simply a result of outsourcing the testing overhead. Working with an experienced testing outsourcer enables organizations to benefit from the supplier’s extensive client-facing experience and the development and refinement over the years of standardized testing processes that have been proved to deliver the greatest benefit in the shortest possible timeframe. Therefore, in many cases, a testing specialist can also provide higher quality testing than a company that simply provides testing as part of a bundle of services. We deploy our methodologies right across the organization as a standard, so that testing execution, whether onshore or offshore, is always effective. The same consistent process is used in all entities, with the same ‘language’ or terminology, utilizing the same process, to provide the same level of quality and end-result. There is, as a result, minimal reinvention of the wheel.

These and other methodologies can improve a company’s current processes either in-house or outsourced and so enable greater operational efficiencies and more accurate measurement throughout the life of the project. They can also help to provide a greater, if not complete, transparency of the testing process and procedure. At any point, an enterprise can ascertain where the project is, the quality of the software being tested and the performance of the project team. This information, allied to quality management tool suites from companies such as IBM, HP, Microsoft and Micro Focus, facilitates testing automation, reporting and collaboration, and becomes extremely valuable to the business particularly in ascertaining and mitigating business risk. Clients may opt to improve and rationalize test processes before outsourcing; this is certainly a ‘belt and braces’ approach but can result in significant expense. Our structured test management approach can be customized and applied to virtually any customer environment, which means outsourcing can be carried out more immediately, and without the expense of improving or creating new testing processes from scratch. Moreover, these models can be adapted to suit the current maturity level of individual clients, meaning this approach is suitable for organizations new to testing, while still providing value to organizations with a higher level of in-house testing expertise.

Quality Through the Discipline of Performance Measures

The discipline of outsourcing—supplier selection, contract and vendor management, development of a formalized business case and adoption of structured processes—paves the way for an organization to take a more integrated approach towards the use of Key Performance Indicators (KPIs). These measure and monitor future testing performance—in essence, how those baseline costs are improved by outsourcing. In particular, clients should measure the benefit of outsourced testing in terms of four key areas: time spent on testing, cost of providing testing resources, time to market and overall application quality. Although every organization is different, KPIs will commonly focus on factors such as employee time spent on testing, the cost of providing testing resources, time to market and defect identification. Using KPIs enables us to unambiguously demonstrate the changes that are being achieved—most notably costs saved. A series of jointly-agreed KPIs enabled us to prove to a major European telecom company that the process improvements and outsourcing had saved them 25% on testing costs over two years. To provide clients with confidence in these outcomes, we often approach contracts from a shared risk/reward perspective. Our experience has shown that this approach enables us to collaborate with the client most effectively on performance metrics and service level agreements. In addition, a collaborative relationship means that if problems arise, the focus isn’t on finding someone to blame—it’s on fixing the problem and getting the project back on track.

Conclusion and Recommendations

Software testing represents perhaps one of the last major opportunities for businesses to reduce IT overheads and improve efficiency through outsourcing. Many of the traditional barriers associated with software testing (ill-defined testing and requirements processes, limited understanding of actual testing costs) can be overcome by selecting an experienced testing partner with proven industry expertise and structured processes that will ensure best practices are consistently applied. When successfully executed, outsourcing of software testing can reduce the cost of the testing function by 50% or more, as well as improving time to market and overall software quality. This is vital because so many businesses now heavily rely on IT applications functioning faultlessly, time after time. There are a number of challenges associated with offshore software testing, and CIOs should not assume that all outsourcing will deliver the same benefits. But to ensure the best possible results from software testing outsourcing, CIOs must remember:

• To hire specialist companies or teams to separately conduct software development and testing;
• To invest time upfront in benchmarking current performance through a thorough business case, ensuring the testing provider delivers a tangible return on investment, and;
• To look for experience in working with companies in many environments and into well-developed methodologies rather than just specific-sector experience.

With these elements in place, the CIO can be reassured that testing is truly delivering business value.