Search-Enhanced Testing

The prime obstacle to automated defect testing has always been the generation of “correct” results against which to judge the behavior of the system under test – the “oracle problem”. So called “back-to-back” testing techniques that exploit the availability of multiple versions of a system to solve the oracle problem have mainly been restricted to very special, safety critical domains such as military and space applications because it is too expensive to manually develop the additional versions.

However, a new generation of software search engines that can find multiple copies of software components at virtually zero cost promise to change this situation. They make it economically feasible to use the knowledge locked in reusable software components to dramatically improve the efficiency of the software testing process.

The process of search-enhanced testing invented by our group is not meant to replace traditional testing techniques but to complement them. For one thing, the approach is currently limited to the relative simple kinds of executable components that can be found and harvested using software search engines like (i.e. largely to unit testing). Nevertheless, our research hypothesis is that, in general, a typical developer, for a given level of effort, will be able to discover more errors through search-enhanced testing than using normal testing approaches. Or alternatively stated, a typical developer will need less time to uncover a given number of errors using search-enhanced testing than using traditional approaches.

Although it may seem paradoxical to use harvested components that already provide the desired functionality to merely help test new, self written components, another advantage of the SET approach is that it provides a way to exploit the knowledge bound up in software repositories. Unfortunately, various obstacles usually stand in the way of the direct reuse of harvested components in new applications – sometimes it is just the “not invented here syndrome” that causes the problem, but often it is more serious legal concerns related to the use of components that are not self-owned (e.g. open source) or the liability issues that arise when such a component fails.

Similar to regression testing, search-enhanced testing allows the quality of new software to be improved using the knowledge embedded in existing software. However, in the case of search-enhanced testing it is embedded in third party components that can be harvested from diverse sources on the Internet, while in regression testing it is locked in previous, self-written tests for a component.

Contact: Werner Janjic