Reproducing Test-Comp Results

A description on how we make the results of the competition reproducible can be found in the competition report; the components that are listed below in the table are described in Sect. 4 "Reproducibility".

Here is the list of components that were used for Test-Comp 2021:

Repositories

Component Repository Release Tag
Test-Generator Archives git@gitlab.com:sosy-lab/test-comp/archives-2022.git testcomp22
Test-Generation Tasks git@gitlab.com:sosy-lab/benchmarking/sv-benchmarks.git testcomp22
Benchmark Definitions git@gitlab.com:sosy-lab/test-comp/bench-defs.git testcomp22
Tool-Info Modules git@github.com:sosy-lab/benchexec.git 3.10
Test-Suite Format git@gitlab.com:sosy-lab/test-comp/test-format testcomp22
BenchExec git@github.com:sosy-lab/benchexec.git 3.10

Reproduction Archives

Content Reproduction Archives
Test-Generation Tasks doi:10.5281/zenodo.5831003
Competition Results doi:10.5281/zenodo.5831012
Test-Suite Generators doi:10.5281/zenodo.5959598
Test Suites (Witnesses) doi:10.5281/zenodo.5831010
BenchExec doi:10.5281/zenodo.5720267

Verifiers

Tool Lang. Jury member Affiliation Archive Bechmark definition Publication
Overview Chair Dirk Beyer LMU Munich, Germany
CMA-ES Fuzz C Hors Concours --, -- cmaesfuzz.zip cmaesfuzz.xml ...
CoVeriTest C Marie-Christine Jakobs TU Darmstadt, Germany coveritest.zip coveritest.xml ...
FuSeBMC C Kaled Alshmrany University of Manchester / Institute of Public Administration, UK / Saudi Arabia FuSeBMC.zip FuSeBMC.xml ...
HybridTiger C Hors Concours --, -- hybridtiger.zip hybridtiger.xml ...
KLEE C Hors Concours --, -- klee.zip klee.xml ...
Legion C Gidon Ernst LMU Munich, Germany legion.zip legion.xml ...
Legion/SymCC C Gidon Ernst LMU Munich, Germany legion-symcc.zip legion-symcc.xml ...
LibKluzzer C Hoang M. Le University of Bremen, Germany libkluzzer.zip libkluzzer.xml ...
PRTest C Thomas Lemberger LMU Munich, Germany prtest.zip prtest.xml ...
Symbiotic C Marek Chalupa Masaryk University, Brno, Czechia symbiotic.zip symbiotic.xml ...
TracerX C Joxan Jaffar National University of Singapore, Singapore tracerx.zip tracerx.xml ...
VeriFuzz C Raveendra Kumar Medicherla Tata Consultancy Services, India verifuzz.zip verifuzz.xml ...

Validators

Tool Lang. Type Contact Affiliation Bechmark definition Publication
TestCov C testcovvalidate-test-suites Thomas Lemberger LMU Munich, Germany testcov-validate-test-suites.xml ...