A Survey on Test Oracles

Abstract

Oracles play a key role in software testing since they state the correctness of the software under test. Unfortunately, many testers tend to judge their own executions, but human oracles are error-prone, slow, and very expensive. In contrast, automated oracles are usually faster, cheaper, and much more reliable: many proposals have addressed the problem, but no unique solution has emerged so far. The variety of these solutions motivated the survey of the state of the art on test oracles presented in this paper. Besides introducing the general concepts, we propose a classification based on the origin of the information used by the oracle. For each oracle type, the survey discusses it, proposes some examples, and highlights its limitations. Moreover, it compares the different types and provides a final holistic assessment. The goal is to provide software testing professionals and researchers with a taxonomy, a critical overview, and a fair assessment of the state of the art in the domain.