5th Intl. Competition on Software Testing held at FASE 2023 in Paris, France.

2023 Competition Report


Tool competitions are a special form of comparative evaluation, where each tool has a team of developers or supporters associated that makes sure that the tool shows its best possible performance. Tool competitions have been a driving force for the development of mature tools that represent the state of the art in several research areas. This web site describes the competition on automatic software testing, which is in 2019 held as a satellite event for the conference TACAS 2019, as part of the TOOLympics event.

There are several new and powerful tools for automatic software testing around, but they are very difficult to compare. The reason is that so far no widely distributed benchmark suite of testing tasks was available and most concepts are only validated in research prototypes. This competition wants to change this: The goal is to establish a set of test tasks for comparing software testers, and the tools are publicized on the Test-Comp web site.

Only few projects aim at producing stable tools that can be used by people outside the respective development groups, and the development of such tools is not continuous. Also, PhD students and PostDocs do not adequately benefit from tool development because theoretical papers count more than papers that present technical and engineering contributions, like tool papers. Through its visibility, this competition wants to change this, showing off the latest implementations of the research results in our community, and give credits to researchers and students who spend considerable amounts of time developing test algorithms and software packages.

Goals of the Competition


One test run for a test generator gets as input (i) a program from the benchmark suite and (ii) a test specification (find bug, or coverage criterion), and returns as output a test suite (i.e., a set of test vectors). The test generator is contributed by the competition participant. The test runs are executed centrally by the competition organizer. The test validator takes as input the test suite and validates it by executing the program on all test vectors: for bug finding it checks if the bug is exposed and for coverage it reports the coverage using gcov. The picture below might help getting an idea of the process.


For questions about the competition, this web page, the benchmarks, or the organization of the competition, please contact the competition chair: Dirk Beyer, LMU Munich, Germany.

For discussion of topics of broad interest that are related to the competition on software testing, please consider posting on our mailing list: test-comp@googlegroups.com.
Web archive: https://groups.google.com/forum/#!forum/test-comp


Test-Comp 2023 is sponsored by:

Logo LMU