Reproducing Test-Comp Results

A description on how we make the results of the competition reproducible can be found in the competition report; the components that are listed below in the table are described in Sect. 4 "Reproducibility".

Here is the list of components that were used for Test-Comp 2021:

Repositories

Component Repository Release Tag
Test-Generator Archives git@gitlab.com:sosy-lab/test-comp/archives-2021.git testcomp21
Test-Generation Tasks git@github.com:sosy-lab/sv-benchmarks.git testcomp21
Benchmark Definitions git@gitlab.com:sosy-lab/test-comp/bench-defs.git testcomp21
Tool-Info Modules git@github.com:sosy-lab/benchexec.git 3.6
Test-Suite Format git@gitlab.com:sosy-lab/software/test-format testcomp21
BenchExec git@github.com:sosy-lab/benchexec.git 3.6

Reproduction Archives

Content Reproduction Archives
Test-Generation Tasks doi:10.5281/zenodo.4459132
Competition Results doi:10.5281/zenodo.4459470
Test Suites (Witnesses) doi:10.5281/zenodo.4459466
BenchExec doi:10.5281/zenodo.4317433

Participating Systems

Tool Lang. Jury member Affiliation Archive Bechmark definition System description
Overview Dirk Beyer LMU Munich, Germany
CMA-ES Fuzz C Gidon Ernst LMU Munich, Germany Download cmaesfuzz.xml ...
CoVeriTest C Marie-Christine Jakobs TU Darmstadt, Germany Download coveritest.xml ...
FuSeBMC C Kaled Alshmrany University of Manchester, UK Download FuSeBMC.xml ...
HybridTiger C Sebastian Ruland TU Darmstadt, Germany Download hybridtiger.xml ...
KLEE C Martin Nowack Imperial College London, UK Download klee.xml ...
Legion C Dongge Liu University of Melbourne, Australia Download legion.xml ...
LibKluzzer C Hoang M. Le University of Bremen, Germany Download libkluzzer.xml ...
PRTest C Thomas Lemberger LMU Munich, Germany Download prtest.xml ...
Symbiotic C Marek Chalupa Masaryk University, Brno, Czechia Download symbiotic.xml ...
TracerX C Joxan Jaffar National University of Singapore, Singapore Download tracerx.xml ...
VeriFuzz C Raveendra Kumar Medicherla Tata Consultancy Services, India Download verifuzz.xml ...

Validators

Validator Contact Affiliation Archive Benchmark definition
TestVal Test-Suite Validator Thomas Lemberger LMU Munich, Germany Download testcov-validate-test-suites.xml