FASE 2022
4th Competition on Software Testing (Test-Comp 2022)

Results of the Competition

This web page presents the results of Test-Comp 2022 - 4th International Competition on Software Testing.

• Competition Report for Test-Comp 2022
• The benchmarks and rules are also available from the competition web site.

The background color is gold for the winner, silver for the second, and bronze for the third.

Here some brief directions for reading the score-based quantile plots:

• A data point (x, y) tells you that the quantile of cumulative score x can be achieved with a minimum of y tasks.
• The right end of the graph displays the achieved score. The score is the competition's definition of quality. You can read the ranking on the top from right to left.
• More information on what to learn from and how to interpret score-based quantile plots can be found in the SV-COMP '15 report on pages 12 and 13.

Here some brief directions for navigating in the BenchExec-generated tables with the results:

• Please note the header with the menu "Select Columns, Filter Rows, etc.", which loads only after all data are retrieved from the server.
• The tables contain all information that I thought you could be interested in. If there are too many columns, you can get rid of some with "Select Columns". You can also investigate special classes of rows using "Filter Rows".
• If you want to see more data, including the time and memory needed for validation, please click on the score value in the table below (works only for base categories) to browse the most detailed tables, which also contain links to obtain the test-suites that the test-generator generated.
• In all results tables, the sources of the verification task is available via click on "prog". The log files are available via click on the status (result).
• If you want to investigate even more, you can download the Test-Comp 2022 results archive, which contains all data that I obtained with BenchExec and from which I compiled all the tables and aggregated data displays. For the most detailed tables, which end on ".xml.bz2.table.html", you can obtain the raw data by accessing the URL with the ".table.html" removed.
• More on table generation is described here.
• Column "score" shows the (not normalized) score assigned to the result of the task.
• Column "test-suite" contains a link to the test-suite archive for download/display.
• Column "inspect test-suite" opens a new page that displays information about the test-suite an a list of test-suites that other test-generators produced for the same program.
• Column "cpu" shows the consumed CPU time.
• Column "wall" shows the elapsed wall time.
• Column "energy" shows the energy consumed by the CPU.
• Column "mem" shows the peak memory usage.
• Column "blkio-w" shows the accumulated amount of information written to block devices.
• Column "blkio-r" shows the accumulated amount of information read from block devices.
• More information on the results are available in the FAQ.

Ranking by Category (with Score-Based Quantile Plots)

Cover-Error 1. FuSeBMC 2. VeriFuzz 3. LibKluzzer	Cover-Branches 1. FuSeBMC 2. VeriFuzz 3. LibKluzzer	Overall 1. FuSeBMC 2. VeriFuzz 3. LibKluzzer

Table of All Results

In every table cell for competition results, we list the points in the first row and the CPU time (rounded to two significant digits) for successful runs in the second row.

The entry 'Hors Concours' in the row for 'Representing Jury Member' means that the tool was added at the organizer's disposition and does not participate in the rankings or prize allocation.
The entry '–' means that the competition candidate opted-out in the category.
The definition of the scoring schema and the categories is given on the respective SV-COMP web pages.

Hide base categories

Participants	Plots	CMA-ES Fuzz	CoVeriTest	FuSeBMC	HybridTiger	KLEE	Legion	Legion/SymCC	LibKluzzer	PRTest	Symbiotic	TracerX	VeriFuzz
Representing Jury Member		Hors Concours	Marie-Christine Jakobs	Kaled Alshmrany	Hors Concours	Hors Concours	Gidon Ernst	Gidon Ernst	Hoang M. Le	Thomas Lemberger	Marek Chalupa	Joxan Jaffar	Raveendra Kumar Medicherla
Affiliation		--, --	TU Darmstadt, Germany	University of Manchester / Institute of Public Administration, UK / Saudi Arabia	--, --	--, --	LMU Munich, Germany	LMU Munich, Germany	University of Bremen, Germany	LMU Munich, Germany	Masaryk University, Brno, Czechia	National University of Singapore, Singapore	Tata Consultancy Services, India
Cover-Error 776 tasks		0	423	628	355	500	57		528	145	463	0	623
CPU time		0 s	120000 s	78000 s	260000 s	11000 s	64000 s		500000 s	200000 s	22000 s	0 s	13000 s
coverage-error-call.ReachSafety-Arrays 100 tasks		0	73	99	69	89	67		97	37	74	0	99
CPU time		0 s	13000 s	21000 s	41000 s	120 s	60000 s		30000 s	4900 s	970 s	0 s	950 s
coverage-error-call.ReachSafety-BitVectors 10 tasks		0	8	10	6	9	0		10	5	8	0	10
CPU time		0 s	300 s	890 s	2300 s	5.4 s	0 s		9000 s	4500 s	160 s	0 s	130 s
coverage-error-call.ReachSafety-ControlFlow 32 tasks		0	18	32	16	27	0		27	0	24	0	30
CPU time		0 s	2500 s	1800 s	12000 s	8.1 s	0 s		20000 s	0 s	880 s	0 s	540 s
coverage-error-call.ReachSafety-ECA 18 tasks		0	3	13	1	13	0		11	0	14	0	15
CPU time		0 s	460 s	3300 s	900 s	370 s	0 s		9900 s	0 s	1200 s	0 s	1300 s
coverage-error-call.ReachSafety-Floats 33 tasks		0	25	33	23	6	0		30	3	0	0	32
CPU time		0 s	3000 s	3800 s	5600 s	2.4 s	0 s		27000 s	2700 s	0 s	0 s	560 s
coverage-error-call.ReachSafety-Heap 56 tasks		0	49	53	43	52	3		53	13	53	0	53
CPU time		0 s	14000 s	1900 s	12000 s	170 s	15 s		35000 s	9700 s	340 s	0 s	390 s
coverage-error-call.ReachSafety-Loops 157 tasks		0	75	146	53	95	4		136	102	81	0	142
CPU time		0 s	38000 s	31000 s	32000 s	3400 s	3600 s		120000 s	91000 s	830 s	0 s	3500 s
coverage-error-call.ReachSafety-ProductLines 169 tasks		0	160	169	53	169	34		169	92	159	0	169
CPU time		0 s	40000 s	1000 s	48000 s	81 s	180 s		150000 s	83000 s	3600 s	0 s	1600 s
coverage-error-call.ReachSafety-Recursive 20 tasks		0	7	19	5	16	0		17	1	17	0	16
CPU time		0 s	6300 s	900 s	710 s	8.7 s	0 s		15000 s	900 s	230 s	0 s	610 s
coverage-error-call.ReachSafety-Sequentialized 107 tasks		0	61	102	92	86	0		81	0	79	0	104
CPU time		0 s	5200 s	11000 s	80000 s	4200 s	0 s		73000 s	0 s	4800 s	0 s	1300 s
coverage-error-call.ReachSafety-XCSP 59 tasks		0	50	52	52	37	0		5	0	41	0	55
CPU time		0 s	2400 s	1200 s	23000 s	2600 s	0 s		4500 s	0 s	8500 s	0 s	1600 s
coverage-error-call.SoftwareSystems-BusyBox-MemSafety 13 tasks		0	0	2	0	1	0		0	0	0	0	2
CPU time		0 s	0 s	300 s	0 s	1.8 s	0 s		0 s	0 s	0 s	0 s	100 s
coverage-error-call.SoftwareSystems-DeviceDriversLinux64-ReachSafety 2 tasks		0	0	0	0	0	0		0	0	0	0	0
CPU time		0 s	0 s	0 s	0 s	0 s	0 s		0 s	0 s	0 s	0 s	0 s
Cover-Branches 3460 tasks		624	1860	2104	1406	1242	1033	1487	1990	896	1802	1746	2075
CPU time		1500000 s	2000000 s	3100000 s	1900000 s	1100000 s	2700000 s	3000000 s	2700000 s	2600000 s	1900000 s	440000 s	3100000 s
coverage-branches.ReachSafety-Arrays 400 tasks		159	257	328	247	104	210	263	323	160	250	243	307
CPU time		190000 s	340000 s	360000 s	310000 s	60000 s	300000 s	330000 s	180000 s	170000 s	310000 s	46000 s	340000 s
coverage-branches.ReachSafety-BitVectors 62 tasks		26	49	49	16	33	34	45	48	33	49	49	48
CPU time		35000 s	21000 s	56000 s	7800 s	22000 s	44000 s	55000 s	56000 s	55000 s	35000 s	14000 s	56000 s
coverage-branches.ReachSafety-ControlFlow 67 tasks		5	36	43	14	25	17	41	40	6	43	39	44
CPU time		46000 s	44000 s	59000 s	41000 s	26000 s	59000 s	60000 s	44000 s	21000 s	30000 s	3800 s	58000 s
coverage-branches.ReachSafety-ECA 29 tasks		0	6	11	2	7	3	3	10	2	10	8	12
CPU time		23000 s	24000 s	26000 s	23000 s	26000 s	19000 s	25000 s	26000 s	26000 s	21000 s	7700 s	25000 s
coverage-branches.ReachSafety-Floats 226 tasks		53	113	122	92	19	72	62	103	46	55	56	119
CPU time		120000 s	88000 s	200000 s	76000 s	9800 s	190000 s	200000 s	200000 s	200000 s	16000 s	5400 s	200000 s
coverage-branches.ReachSafety-Heap 143 tasks		23	100	104	84	93	81	78	104	49	98	101	101
CPU time		69000 s	89000 s	130000 s	57000 s	71000 s	110000 s	130000 s	110000 s	82000 s	75000 s	11000 s	130000 s
coverage-branches.ReachSafety-Loops 727 tasks		211	574	591	467	380	357	542	575	359	538	544	587
CPU time		290000 s	450000 s	650000 s	390000 s	210000 s	550000 s	650000 s	600000 s	640000 s	300000 s	73000 s	650000 s
coverage-branches.ReachSafety-ProductLines 263 tasks		19	77	77	56	74	70	74	77	48	69	77	77
CPU time		94000 s	78000 s	200000 s	220000 s	10000 s	190000 s	220000 s	240000 s	240000 s	6200 s	11000 s	240000 s
coverage-branches.ReachSafety-Recursive 53 tasks		25	41	45	39	21	26	27	43	11	45	40	41
CPU time		28000 s	48000 s	48000 s	29000 s	21000 s	21000 s	47000 s	45000 s	14000 s	43000 s	6600 s	45000 s
coverage-branches.ReachSafety-Sequentialized 103 tasks		0	79	90	58	35	1	43	75	11	51	57	91
CPU time		120 s	82000 s	92000 s	77000 s	26000 s	93000 s	93000 s	92000 s	93000 s	43000 s	18000 s	93000 s
coverage-branches.ReachSafety-XCSP 119 tasks		0	116	107	119	102	2	102	118	102	114	96	110
CPU time		180 s	21000 s	110000 s	6000 s	17000 s	110000 s	100000 s	110000 s	110000 s	44000 s	9500 s	110000 s
coverage-branches.ReachSafety-Combinations 671 tasks		63	238	401	167	196	179	224	292	79	338	295	351
CPU time		320000 s	560000 s	600000 s	500000 s	470000 s	520000 s	590000 s	610000 s	600000 s	490000 s	180000 s	600000 s
coverage-branches.SoftwareSystems-BusyBox-MemSafety 75 tasks		0	12	25	6	21	0	0	24	15	19	18	29
CPU time		24000 s	67000 s	62000 s	60000 s	62000 s	68000 s	68000 s	64000 s	66000 s	63000 s	18000 s	68000 s
coverage-branches.SoftwareSystems-DeviceDriversLinux64-ReachSafety 290 tasks		13	60	59	6	25	56	47	57	16	42	56	57
CPU time		190000 s	40000 s	260000 s	28000 s	60000 s	260000 s	260000 s	230000 s	250000 s	240000 s	13000 s	260000 s
coverage-branches.SoftwareSystems-SQLite-MemSafety 1 tasks		0	0	0	0	0	0	0	0	0	0	0	0
CPU time		880 s	900 s	900 s	900 s	30 s	900 s	190 s	130 s	900 s	900 s	48 s	900 s
coverage-branches.Termination-MainHeap 231 tasks		143	212	213	195	118	168	145	204	60	179	192	202
CPU time		100000 s	26000 s	210000 s	22000 s	8400 s	190000 s	210000 s	130000 s	68000 s	190000 s	24000 s	200000 s
Overall 4236 tasks		382	2293	3003	1830	2125	787		2658	945	2367	1069	2971
CPU time		1500000 s	2100000 s	3100000 s	2100000 s	1100000 s	2800000 s		3200000 s	2800000 s	1900000 s	440000 s	3100000 s
Participants	Plots	CMA-ES Fuzz	CoVeriTest	FuSeBMC	HybridTiger	KLEE	Legion	Legion/SymCC	LibKluzzer	PRTest	Symbiotic	TracerX	VeriFuzz

Note on meta-categories: The score is not the sum of scores of the sub-categories (normalization). The run time is the sum of run times of the sub-categories, rounded to two significant digits.

If you did not find what you are looking for, please do not hesitate to contact Dirk Beyer (competition chair).