1st Competition on Software Testing (Test-Comp)

Results of the Competition

This web page presents the results of the 2019 1st International Competition on Software Teesting (Test-Comp '19).

The background color is gold for the winner, silver for the second, and bronze for the third.

Here some brief directions for reading the score-based quantile plots:

Here some brief directions for navigating in the BenchExec-generated tables with the results:


Ranking by Category (with Score-Based Quantile Plots)

What you can learn from a score-based quantile plot and how to interpret it, is described in the competition report on pages 12 and 13.

Cover-Error
1. VeriFuzz
2. KLEE
3. CoVeriTest
Quantile-Plot
Cover-Branches
1. VeriFuzz
2. KLEE
3. CoVeriTest
Quantile-Plot
Overall
1. VeriFuzz
2. KLEE
3. CoVeriTest
Quantile-Plot

Table of All Results

In every table cell for competition results, we list the points in the first row and the CPU time (rounded to two significant digits) for successful runs in the second row.

The entry '–' means that the competition candidate opted-out in the category.
The definition of the scoring schema and the categories is given on the respective Test-Comp web pages.

ToolsPlotsCoVeriTestCPA/Tiger-MGPESBMC-bkind 6.1ESBMC-falsif 6.1FairFuzzKLEEPRTestSymbioticVeriFuzz
Representing Jury MemberMarie-Christine JakobsSebastian RulandRafael MenezesMikhail GadelhaCaroline LemieuxCristian CadarThomas LembergerMartina Vitovsk√°Raveendra Kumar Medicherla
AffiliationLMU Munich, GermanyTU Darmstadt, GermanyFederal University of AmazonasUniversity of SouthamptonUniversity of California, BerkeleyImperial College LondonLMU Munich, GermanyMasaryk University, CzechiaTCS, India
Cover-Error
636 tasks
Quantile-Plot 397361237247365499193365595
CPU time31000 s17000 s40000 s22000 s150000 s18000 s3900 s25000 s53000 s
coverage-error-call.ReachSafety-Arrays
22 tasks
Quantile-Plot 1166161851820
CPU time23 s3.3 s190 s190 s9800 s45 s1.3 s250 s510 s
coverage-error-call.ReachSafety-BitVectors
10 tasks
Quantile-Plot 7766895510
CPU time150 s59 s1.6 s1.4 s2100 s2.0 s9.0 s6.2 s880 s
coverage-error-call.ReachSafety-ControlFlow
8 tasks
Quantile-Plot 88387
CPU time58 s54 s1300 s1.5 s430 s
coverage-error-call.ReachSafety-ECA
412 tasks
Quantile-Plot 195108251254132274255376
CPU time23000 s12000 s40000 s22000 s120000 s14000 s23000 s45000 s
coverage-error-call.ReachSafety-Floats
32 tasks
Quantile-Plot 2523661124130
CPU time2500 s1700 s.97 s.95 s4700 s1800 s.54 s1400 s
coverage-error-call.ReachSafety-Heap
9 tasks
Quantile-Plot 991199588
CPU time130 s48 s.10 s.10 s140 s1.4 s2000 s5.7 s35 s
coverage-error-call.ReachSafety-Loops
27 tasks
Quantile-Plot 222117191824172326
CPU time620 s150 s18 s12 s6100 s43 s180 s35 s150 s
coverage-error-call.ReachSafety-Recursive
16 tasks
Quantile-Plot 31516151611615
CPU time11 s19 s770 s3900 s34 s.23 s8.8 s1400 s
coverage-error-call.ReachSafety-Sequentialized
100 tasks
Quantile-Plot 61617946100
CPU time4300 s3000 s3200 s1500 s3200 s
Cover-Branches
1720 tasks
Quantile-Plot 11539660087412264769071238
CPU time1200000 s1200000 s48 s49 s840000 s1100000 s1400000 s990000 s1500000 s
coverage-branches.ReachSafety-Arrays
169 tasks
Quantile-Plot 85760012412042116130
CPU time140000 s110000 s4.6 s4.9 s57000 s34000 s90000 s69000 s150000 s
coverage-branches.ReachSafety-BitVectors
32 tasks
Quantile-Plot 2422002124132125
CPU time9400 s15000 s.89 s.89 s9400 s10000 s23000 s12000 s29000 s
coverage-branches.ReachSafety-ControlFlow
19 tasks
Quantile-Plot 14140061411214
CPU time990 s1600 s.49 s.58 s5100 s920 s17000 s31 s17000 s
coverage-branches.ReachSafety-ECA
1046 tasks
Quantile-Plot 36016400466721860762
CPU time890000 s940000 s29 s30 s670000 s940000 s940000 s790000 s930000 s
coverage-branches.ReachSafety-Floats
212 tasks
Quantile-Plot 10499008354845090
CPU time79000 s41000 s5.8 s5.9 s60000 s20000 s180000 s19000 s190000 s
coverage-branches.ReachSafety-Heap
25 tasks
Quantile-Plot 1919001619131818
CPU time2300 s1000 s.72 s.69 s480 s900 s19000 s11 s22000 s
coverage-branches.ReachSafety-Loops
117 tasks
Quantile-Plot 9281007393607884
CPU time57000 s40000 s3.2 s3.4 s31000 s59000 s79000 s55000 s100000 s
coverage-branches.ReachSafety-Recursive
38 tasks
Quantile-Plot 291200303382531
CPU time8600 s110 s1.0 s1.1 s10000 s31000 s9000 s23000 s34000 s
coverage-branches.ReachSafety-Sequentialized
62 tasks
Quantile-Plot 55490004952849
CPU time49000 s48000 s1.8 s1.8 s200 s26000 s56000 s18000 s55000 s
Overall
2356 tasks
Quantile-Plot 152413314384571275176468312981951
CPU time1300000 s1200000 s40000 s23000 s990000 s1100000 s1400000 s1000000 s1600000 s
ToolsPlotsCoVeriTestCPA/Tiger-MGPESBMC-bkind 6.1ESBMC-falsif 6.1FairFuzzKLEEPRTestSymbioticVeriFuzz

Note on meta-categories: The score is not the sum of scores of the sub-categories (normalization). The run time is the sum of run times of the sub-categories, rounded to two significant digits.