FASE 2021
3rd Competition on Software Testing (Test-Comp 2021)

Results of the Competition

This web page presents the results of Test-Comp 2021 - 3rd International Competition on Software Testing.

• Competition Definition
• Competition Report for Test-Comp 2021
• The benchmarks and rules are also available from the competition web site.

The background color is gold for the winner, silver for the second, and bronze for the third.

Here some brief directions for reading the score-based quantile plots:

• A data point (x, y) tells you that the quantile of cumulative score x can be achieved with a minimum of y tasks.
• The right end of the graph displays the achieved score. The score is the competition's definition of quality. You can read the ranking on the top from right to left.
• More information on what to learn from and how to interpret score-based quantile plots can be found in the SV-COMP '15 report on pages 12 and 13.

Here some brief directions for navigating in the BenchExec-generated tables with the results:

• Please note the header with the menu "Select Columns, Filter Rows, etc.", which loads only after all data are retrieved from the server.
• The tables contain all information that I thought you could be interested in. If there are too many columns, you can get rid of some with "Select Columns". You can also investigate special classes of rows using "Filter Rows".
• If you want to see more data, including the time and memory needed for validation, please click on the score value in the table below (works only for base categories) to browse the most detailed tables, which also contain links to obtain the test-suites that the test-generator generated.
• In all results tables, the sources of the verification task is available via click on "prog". The log files are available via click on the status (result).
• If you want to investigate even more, you can download the Test-Comp 2021 results archive, which contains all data that I obtained with BenchExec and from which I compiled all the tables and aggregated data displays. For the most detailed tables, which end on ".xml.bz2.table.html", you can obtain the raw data by accessing the URL with the ".table.html" removed.
• More on table generation is described here.
• Column "score" shows the (not normalized) score assigned to the result of the task.
• Column "test-suite" contains a link to the test-suite archive for download/display.
• Column "inspect test-suite" opens a new page that displays information about the test-suite an a list of test-suites that other test-generators produced for the same program.
• Column "cpu" shows the consumed CPU time.
• Column "wall" shows the elapsed wall time.
• Column "energy" shows the energy consumed by the CPU.
• Column "mem" shows the peak memory usage.
• Column "blkio-w" shows the accumulated amount of information written to block devices.
• Column "blkio-r" shows the accumulated amount of information read from block devices.
• More information on the results are available in the FAQ.

Ranking by Category (with Score-Based Quantile Plots)

Cover-Error 1. FuSeBMC 2. VeriFuzz 3. LibKluzzer	Cover-Branches 1. VeriFuzz 2. LibKluzzer 3. Symbiotic	Overall 1. VeriFuzz 2. FuSeBMC 3. LibKluzzer

Table of All Results

In every table cell for competition results, we list the points in the first row and the CPU time (rounded to two significant digits) for successful runs in the second row.

The entry '–' means that the competition candidate opted-out in the category.
The definition of the scoring schema and the categories is given on the respective SV-COMP web pages.

Hide base categories

Participants	Plots	FuSeBMC	CMA-ES Fuzz	CoVeriTest	HybridTiger	KLEE	Legion	LibKluzzer	PRTest	Symbiotic	TracerX	VeriFuzz
Representing Jury Member		Kaled Alshmrany	Gidon Ernst	Marie-Christine Jakobs	Sebastian Ruland	Martin Nowack	Dongge Liu	Hoang M. Le	Thomas Lemberger	Marek Chalupa	Joxan Jaffar	Raveendra Kumar M
Affiliation		University of Manchester, UK	LMU Munich, Germany	TU Darmstadt, Germany	TU Darmstadt, Germany	Imperial College London, UK	University of Melbourne, Australia	University of Bremen, Germany	LMU Munich, Germany	Masaryk University Brno, Czechia	National University of Singapore, Singapore	Tata Consultancy Services, India
Cover-Error 607 tasks		405	0	225	266	339	35	359	79	314	246	385
CPU time		78000 s	0 s	18000 s	200000 s	11000 s	64000 s	320000 s	110000 s	20000 s	46000 s	9300 s
coverage-error-call.ReachSafety-Arrays 100 tasks		93	0	59	69	88	67	96	11	73	75	95
CPU time		19000 s	0 s	2000 s	41000 s	130 s	60000 s	29000 s	4600 s	99 s	3800 s	500 s
coverage-error-call.ReachSafety-BitVectors 10 tasks		10	0	8	6	9	0	9	5	8	7	9
CPU time		1400 s	0 s	290 s	2300 s	5.4 s	0 s	8100 s	4500 s	140 s	2100 s	200 s
coverage-error-call.ReachSafety-ControlFlow 32 tasks		8	0	8	8	10	0	11	0	7	9	9
CPU time		950 s	0 s	270 s	4900 s	3.3 s	0 s	7500 s	0 s	170 s	280 s	380 s
coverage-error-call.ReachSafety-ECA 18 tasks		8	0	2	1	14	0	11	0	15	2	16
CPU time		1400 s	0 s	450 s	900 s	490 s	0 s	9900 s	0 s	1400 s	1100 s	1600 s
coverage-error-call.ReachSafety-Floats 33 tasks		32	0	16	22	6	0	30	3	0	0	30
CPU time		4500 s	0 s	770 s	4600 s	2.5 s	0 s	27000 s	2700 s	0 s	0 s	490 s
coverage-error-call.ReachSafety-Heap 57 tasks		45	0	37	38	46	0	47	9	47	44	47
CPU time		2300 s	0 s	4700 s	8400 s	170 s	0 s	29000 s	7000 s	450 s	4400 s	280 s
coverage-error-call.ReachSafety-Loops 158 tasks		131	0	35	53	96	4	138	102	82	78	136
CPU time		32000 s	0 s	1300 s	31000 s	3400 s	3600 s	120000 s	91000 s	1500 s	9000 s	2900 s
coverage-error-call.ReachSafety-Recursive 20 tasks		19	0	0	5	16	0	17	1	17	14	13
CPU time		910 s	0 s	0 s	720 s	8.8 s	0 s	15000 s	900 s	210 s	5600 s	310 s
coverage-error-call.ReachSafety-Sequentialized 107 tasks		101	0	61	93	86	0	83	0	79	57	99
CPU time		13000 s	0 s	6100 s	81000 s	4200 s	0 s	75000 s	0 s	4800 s	10000 s	2100 s
coverage-error-call.ReachSafety-XCSP 59 tasks		53	0	46	52	37	0	3	0	41	31	25
CPU time		3100 s	0 s	1700 s	23000 s	2600 s	0 s	2700 s	0 s	11000 s	9700 s	540 s
coverage-error-call.SoftwareSystems-BusyBox-MemSafety 11 tasks		0	0	0	0	0	0	0	0	0	0	0
CPU time		0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s
coverage-error-call.SoftwareSystems-DeviceDriversLinux64-ReachSafety 2 tasks		0	0	0	0	0	0	0	0	0	0	0
CPU time		0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s
Cover-Branches 2566 tasks		1161	411	1128	860	784	651	1292	519	1169	1087	1389
CPU time		1400000 s	1100000 s	1400000 s	1200000 s	760000 s	2000000 s	1900000 s	1800000 s	1600000 s	710000 s	2300000 s
coverage-branches.ReachSafety-Arrays 400 tasks		284	139	229	225	96	195	296	119	226	223	295
CPU time		260000 s	190000 s	340000 s	310000 s	60000 s	300000 s	180000 s	170000 s	310000 s	130000 s	350000 s
coverage-branches.ReachSafety-BitVectors 62 tasks		37	23	39	13	28	29	40	27	37	37	38
CPU time		20000 s	35000 s	24000 s	7700 s	24000 s	44000 s	56000 s	55000 s	36000 s	31000 s	56000 s
coverage-branches.ReachSafety-ControlFlow 67 tasks		15	4	16	3	8	8	16	5	18	15	18
CPU time		47000 s	46000 s	44000 s	39000 s	23 s	59000 s	18000 s	23000 s	3800 s	380 s	59000 s
coverage-branches.ReachSafety-ECA 29 tasks		5	0	6	2	7	3	10	2	10	7	12
CPU time		25000 s	23000 s	24000 s	23000 s	26000 s	19000 s	26000 s	26000 s	22000 s	15000 s	25000 s
coverage-branches.ReachSafety-Floats 226 tasks		103	51	98	84	16	64	90	41	50	48	99
CPU time		72000 s	120000 s	94000 s	77000 s	9900 s	190000 s	200000 s	200000 s	15000 s	13000 s	200000 s
coverage-branches.ReachSafety-Heap 143 tasks		88	19	79	74	81	69	90	40	84	86	86
CPU time		63000 s	71000 s	88000 s	57000 s	71000 s	100000 s	110000 s	83000 s	75000 s	18000 s	130000 s
coverage-branches.ReachSafety-Loops 581 tasks		412	152	402	338	274	271	419	252	383	385	424
CPU time		270000 s	240000 s	350000 s	310000 s	190000 s	450000 s	460000 s	510000 s	280000 s	180000 s	520000 s
coverage-branches.ReachSafety-Recursive 53 tasks		36	19	31	31	18	20	36	9	38	34	35
CPU time		25000 s	28000 s	48000 s	29000 s	21000 s	21000 s	45000 s	14000 s	43000 s	21000 s	47000 s
coverage-branches.ReachSafety-Sequentialized 82 tasks		62	0	61	39	26	1	55	8	36	41	71
CPU time		27000 s	93 s	68000 s	57000 s	23000 s	74000 s	73000 s	74000 s	34000 s	28000 s	74000 s
coverage-branches.ReachSafety-XCSP 119 tasks		97	0	80	80	81	2	80	79	93	69	88
CPU time		27000 s	180 s	5000 s	6000 s	17000 s	110000 s	110000 s	110000 s	110000 s	48000 s	110000 s
coverage-branches.ReachSafety-Combinations 210 tasks		15	0	31	8	82	18	139	2	135	99	180
CPU time		180000 s	34000 s	190000 s	190000 s	190000 s	110000 s	190000 s	190000 s	170000 s	110000 s	190000 s
coverage-branches.SoftwareSystems-BusyBox-MemSafety 72 tasks		1	0	5	4	6	0	6	4	7	4	8
CPU time		11000 s	23000 s	64000 s	58000 s	59000 s	65000 s	61000 s	63000 s	60000 s	31000 s	65000 s
coverage-branches.SoftwareSystems-DeviceDriversLinux64-ReachSafety 290 tasks		35	13	60	6	25	56	58	16	44	56	57
CPU time		260000 s	190000 s	41000 s	28000 s	60000 s	260000 s	230000 s	250000 s	240000 s	24000 s	250000 s
coverage-branches.SoftwareSystems-SQLite-MemSafety 1 tasks		0	0	0	0	0	0	0	0	0	0	0
CPU time		880 s	880 s	900 s	900 s	31 s	900 s	130 s	900 s	900 s	48 s	900 s
coverage-branches.Termination-MainHeap 231 tasks		202	138	193	189	119	166	199	51	178	185	204
CPU time		110000 s	98000 s	24000 s	18000 s	10000 s	190000 s	120000 s	68000 s	190000 s	65000 s	210000 s
Overall 3173 tasks		1776	254	1286	1228	1370	495	1738	526	1543	1315	1865
CPU time		1500000 s	1100000 s	1400000 s	1400000 s	770000 s	2000000 s	2200000 s	1900000 s	1600000 s	760000 s	2300000 s
Participants	Plots	FuSeBMC	CMA-ES Fuzz	CoVeriTest	HybridTiger	KLEE	Legion	LibKluzzer	PRTest	Symbiotic	TracerX	VeriFuzz

Note on meta-categories: The score is not the sum of scores of the sub-categories (normalization). The run time is the sum of run times of the sub-categories, rounded to two significant digits.

If you did not find what you are looking for, please do not hesitate to contact Dirk Beyer (competition chair).