FASE 2024
6th Competition on Software Testing (Test-Comp 2024)

Results of the Competition

This web page presents the results of Test-Comp 2024 - 6th International Competition on Software Testing.

• Competition Report for Test-Comp 2023
• The benchmarks and rules are also available from the competition web site.

The background color is gold for the winner, silver for the second, and bronze for the third.

Here some brief directions for reading the score-based quantile plots:

• A data point (x, y) tells you that the quantile of cumulative score x can be achieved with a minimum of y tasks.
• The right end of the graph displays the achieved score. The score is the competition's definition of quality. You can read the ranking on the top from right to left.
• More information on what to learn from and how to interpret score-based quantile plots can be found in the SV-COMP '15 report on pages 12 and 13.

Here some brief directions for navigating in the BenchExec-generated tables with the results:

• Please note the header with the menu "Select Columns, Filter Rows, etc.", which loads only after all data are retrieved from the server.
• The tables contain all information that I thought you could be interested in. If there are too many columns, you can get rid of some with "Select Columns". You can also investigate special classes of rows using "Filter Rows".
• If you want to see more data, including the time and memory needed for validation, please click on the score value in the table below (works only for base categories) to browse the most detailed tables, which also contain links to obtain the test-suites that the test-generator generated.
• In all results tables, the sources of the verification task is available via click on "prog". The log files are available via click on the status (result).
• If you want to investigate even more, you can download the Test-Comp 2023 results archive, which contains all data that I obtained with BenchExec and from which I compiled all the tables and aggregated data displays. For the most detailed tables, which end on ".xml.bz2.table.html", you can obtain the raw data by accessing the URL with the ".table.html" removed.
• More on table generation is described here.
• Column "score" shows the (not normalized) score assigned to the result of the task.
• Column "test-suite" contains a link to the test-suite archive for download/display.
• Column "inspect test-suite" opens a new page that displays information about the test-suite an a list of test-suites that other test-generators produced for the same program.
• Column "cpu" shows the consumed CPU time.
• Column "wall" shows the elapsed wall time.
• Column "energy" shows the energy consumed by the CPU.
• Column "mem" shows the peak memory usage.
• Column "blkio-w" shows the accumulated amount of information written to block devices.
• Column "blkio-r" shows the accumulated amount of information read from block devices.
• More information on the results are available in the FAQ.

Ranking by Category (with Score-Based Quantile Plots)

Cover-Error 1. FuSeBMC 2. FuSeBMC-AI 3. Symbiotic	Cover-Branches 1. FuSeBMC 2. FuSeBMC-AI 3. Fizzer	Overall 1. FuSeBMC 2. FuSeBMC-AI 3. KLEEF

Table of All Results

In every table cell for competition results, we list the points in the first row and the CPU time (rounded to two significant digits) for successful runs in the second row.

The entry 'Hors Concours' in the row for 'Representing Jury Member' means that the tool was added at the organizer's disposition and does not participate in the rankings or prize allocation.
The entry '–' means that the competition candidate opted-out in the category.
The definition of the scoring schema and the categories is given on the respective SV-COMP web pages.

Hide base categories

Participants	Plots	cetfuzz	CoVeriTest	ESBMC-kind	FDSE	Fizzer	FuSeBMC	FuSeBMC-AI	HybridTiger	KLEE	KLEEF	Legion	Legion/SymCC	Owi	PRTest	Rizzer	Symbiotic	TracerX	TracerX-WP	UTestGen	WASP-C
Representing Jury Member		Sumesh Divakaran	Marie-Christine Jakobs	Hors Concours	Zhenbang Chen	Marek Trtík	Kaled Alshmrany	Mohannad Aldughaim	Hors Concours	Hors Concours	Yurii Kostyukov	Hors Concours	Hors Concours	Léo Andrès	Thomas Lemberger	Adam Štafa	Martin Jonáš	Joxan Jaffar	Joxan Jaffar	Max Barth	Hors Concours
Affiliation		College of Engineering Trivandrum, India	LMU Munich, Germany	--, --	National University of Defense Technology, China	Masaryk University, Brno, Czechia	University of Manchester / Institute of Public Administration, UK / Saudi Arabia	University of Manchester / King Saud University, UK / Saudi Arabia	--, --	--, --	RnD Toolchain Labs, Huawei, China	--, --	--, --	OCamlPro / LMF, France	LMU Munich, Germany	Masaryk University, Brno, Czechia	Masaryk University, Brno, Czechia	National University of Singapore, Singapore	National University of Singapore, Singapore	LMU Munich, Germany	--, --
Cover-Error 1217 tasks		226	462	195	617	583	930	926	393	713	655		264	256	167	555	666	509	322	409	532
CPU time		34000 s	92000 s	3100 s	110000 s	180000 s	270000 s	250000 s	210000 s	27000 s	30000 s		2100 s	3000 s	150000 s	190000 s	19000 s	48000 s	15000 s	36000 s	9400 s
ReachSafety-Arrays 100 tasks		6	73	5	92	82	99	97	69	89	89	67	16	45	37	82	73	74	68	73	89
CPU time		1600 s	8300 s	46 s	3800 s	57000 s	21000 s	19000 s	41000 s	120 s	1700 s	60000 s	15 s	79 s	5000 s	57000 s	74 s	3000 s	150 s	2100 s	1300 s
ReachSafety-BitVectors 9 tasks		3	7	0	8	6	9	9	5	8	9	1	5	5	5	7	8	6	3	3	6
CPU time		810 s	270 s	0 s	3100 s	2600 s	880 s	930 s	1400 s	5.4 s	63 s	270 s	510 s	24 s	4500 s	3500 s	53 s	1000 s	24 s	230 s	290 s
ReachSafety-ControlFlow 5 tasks		3	1	0	3	5	5	5	0	4	4	0	3	1	0	3	5	3	2	1	4
CPU time		2400 s	900 s	0 s	9.8 s	3500 s	500 s	68 s	0 s	2.2 s	1.8 s	0 s	430 s	4.2 s	0 s	1200 s	130 s	1.1 s	2.4 s	650 s	120 s
ReachSafety-ECA 18 tasks		7	3	0	10	5	12	12	1	14	12	0	2	1	0	6	14	3	1	4	2
CPU time		5800 s	1300 s	0 s	9000 s	4300 s	2800 s	2900 s	900 s	550 s	320 s	0 s	12 s	3.2 s	0 s	5200 s	670 s	2000 s	220 s	1300 s	190 s
ReachSafety-Floats 33 tasks		5	24	14	14	18	33	33	22	8	9	0	1	3	3	18	0	0	0	19	23
CPU time		2400 s	2900 s	110 s	400 s	950 s	4000 s	3200 s	4800 s	3.1 s	8.1 s	0 s	0.88 s	9.1 s	2700 s	88 s	0 s	0 s	0 s	2800 s	2800 s
ReachSafety-Fuzzle 15 tasks		0	1	1	1	0	15	15	0	1	1	0	0	0	1	0	1	0	0	14	0
CPU time		0 s	62 s	23 s	33 s	0 s	1900 s	1500 s	0 s	4.2 s	2.2 s	0 s	0 s	0 s	900 s	0 s	25 s	0 s	0 s	3300 s	0 s
ReachSafety-Heap 53 tasks		20	48	6	48	51	51	51	42	50	48	3	46	39	12	50	51	48	44	42	40
CPU time		7300 s	16000 s	1.0 s	9600 s	30000 s	1900 s	310 s	11000 s	200 s	910 s	21 s	830 s	83 s	10000 s	26000 s	300 s	3400 s	200 s	11000 s	410 s
ReachSafety-Loops 125 tasks		66	70	67	110	112	122	122	48	77	85	2	36	68	84	113	75	73	70	80	107
CPU time		6500 s	37000 s	710 s	23000 s	31000 s	31000 s	45000 s	29000 s	1800 s	4000 s	1800 s	37 s	2700 s	75000 s	32000 s	1800 s	7900 s	2800 s	5900 s	710 s
ReachSafety-ProductLines 169 tasks		0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
CPU time		0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s
ReachSafety-Recursive 21 tasks		10	7	0	19	16	20	20	5	17	19	0	17	15	2	17	18	16	10	13	16
CPU time		7300 s	3800 s	0 s	5000 s	2400 s	960 s	1900 s	760 s	9.6 s	910 s	0 s	170 s	50 s	1800 s	3200 s	110 s	3700 s	5400 s	380 s	61 s
ReachSafety-Sequentialized 106 tasks		0	62	44	79	43	102	102	87	86	92	0	0	0	0	51	79	59	43	9	52
CPU time		0 s	5300 s	73 s	51000 s	38000 s	12000 s	13000 s	76000 s	4400 s	5300 s	0 s	0 s	0 s	0 s	45000 s	5400 s	12000 s	850 s	4900 s	1500 s
ReachSafety-XCSP 59 tasks		0	50	54	10	0	52	52	52	37	51	0	0	1	0	0	46	21	29	13	53
CPU time		0 s	2800 s	1900 s	2300 s	0 s	1500 s	890 s	23000 s	2800 s	1900 s	0 s	0 s	2.6 s	0 s	0 s	9400 s	3800 s	5700 s	2200 s	1500 s
ReachSafety-Hardware 497 tasks		26	84	19	28	41	288	278	39	321	342	0	25	0	57	40	49	49	0	12	35
CPU time		59 s	13000 s	180 s	6000 s	13000 s	190000 s	160000 s	24000 s	17000 s	15000 s	0 s	44 s	0 s	51000 s	12000 s	660 s	12000 s	0 s	2000 s	410 s
SoftwareSystems-BusyBox-MemSafety 4 tasks		0	0	0	0	1	1	1	0	1	0	0	0	0	0	0	1	1	0	0	0
CPU time		0 s	0 s	0 s	0 s	870 s	320 s	520 s	0 s	1.8 s	0 s	0 s	0 s	0 s	0 s	0 s	32 s	120 s	0 s	0 s	0 s
SoftwareSystems-OpenBSD-MemSafety 1 tasks		0	0	0	1	1	1	1	0	1	0	0	0	0	0	1	1	1	0	0	0
CPU time		0 s	0 s	0 s	1.4 s	2.4 s	1.4 s	4.8 s	0 s	0.43 s	0 s	0 s	0 s	0 s	0 s	2.4 s	1.3 s	1.8 s	0 s	0 s	0 s
SoftwareSystems-DeviceDriversLinux64-ReachSafety 2 tasks		0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
CPU time		0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s
Cover-Branches 9825 tasks		2197	4826		5132	5146	5478	5418	3987	3023	4975	2896	3381	2241	2980		3957	4435	1521	4195	2838
CPU time		1700000 s	5700000 s		4500000 s	6200000 s	8800000 s	8500000 s	4900000 s	2700000 s	6100000 s	8100000 s	8400000 s	6100000 s	8200000 s		2800000 s	2100000 s	1600000 s	4400000 s	3500000 s
ReachSafety-Arrays 468 tasks		96	316		382	375	389	389	305	152	351	254	187	151	176		301	302	172	300	292
CPU time		150000 s	340000 s		40000 s	290000 s	420000 s	420000 s	320000 s	64000 s	310000 s	360000 s	380000 s	140000 s	170000 s		360000 s	150000 s	160000 s	330000 s	260000 s
ReachSafety-BitVectors 62 tasks		36	49		49	48	49	49	16	32	48	33	45	29	33		46	49	32	45	34
CPU time		35000 s	21000 s		34000 s	41000 s	56000 s	56000 s	7900 s	22000 s	32000 s	52000 s	41000 s	35000 s	55000 s		35000 s	35000 s	18000 s	36000 s	29000 s
ReachSafety-ControlFlow 87 tasks		47	50		53	54	58	58	21	36	56	25	42	19	14		56	53	22	44	45
CPU time		61000 s	48000 s		24000 s	63000 s	77000 s	77000 s	48000 s	41000 s	39000 s	76000 s	72000 s	27000 s	40000 s		40000 s	12000 s	40000 s	56000 s	31000 s
ReachSafety-ECA 30 tasks		8	5		8	7	10	9	2	7	9	3	3	3	2		9	7	0	4	4
CPU time		20000 s	25000 s		26000 s	22000 s	27000 s	27000 s	24000 s	27000 s	25000 s	21000 s	21000 s	24000 s	27000 s		14000 s	15000 s	26000 s	21000 s	25000 s
ReachSafety-Floats 777 tasks		40	200		477	493	449	438	439	20	162	87	114	26	89		122	95	80	113	63
CPU time		110000 s	310000 s		20000 s	190000 s	700000 s	700000 s	500000 s	15000 s	260000 s	470000 s	680000 s	440000 s	700000 s		23000 s	13000 s	16000 s	380000 s	55000 s
ReachSafety-Fuzzle 15 tasks		0	0		1	0	3	2	0	0	1	0	1	0	2		0	0	0	0	0
CPU time		13000 s	14000 s		7800 s	12000 s	13000 s	13000 s	14000 s	7200 s	3700 s	13000 s	12000 s	13000 s	14000 s		5100 s	11000 s	1.0 s	14000 s	12000 s
ReachSafety-Hardness 3625 tasks		22	2070		2495	2350	2564	2381	2091	738	1933	86	144	58	646		336	390	128	996	121
CPU time		20000 s	2400000 s		2900000 s	3100000 s	3300000 s	3300000 s	1500000 s	760000 s	2800000 s	3300000 s	3300000 s	3100000 s	3300000 s		290000 s	200000 s	300000 s	1000000 s	890000 s
ReachSafety-Hardware 1071 tasks		559	851		682	862	751	760	631	286	698	569	640	1	965		100	866	0	412	16
CPU time		10000 s	780000 s		530000 s	930000 s	960000 s	960000 s	860000 s	680000 s	920000 s	930000 s	920000 s	930000 s	960000 s		230000 s	620000 s	74 s	840000 s	930000 s
ReachSafety-Heap 256 tasks		75	142		152	155	156	155	105	127	152	130	139	95	70		144	151	87	137	101
CPU time		99000 s	130000 s		55000 s	110000 s	230000 s	230000 s	71000 s	93000 s	94000 s	200000 s	220000 s	75000 s	150000 s		96000 s	30000 s	63000 s	120000 s	63000 s
ReachSafety-Loops 776 tasks		278	610		620	614	628	629	498	412	623	376	299	430	382		575	581	433	603	455
CPU time		270000 s	480000 s		92000 s	240000 s	700000 s	700000 s	420000 s	230000 s	300000 s	580000 s	650000 s	250000 s	680000 s		310000 s	190000 s	230000 s	290000 s	220000 s
ReachSafety-ProductLines 263 tasks		0	0		0	0	0	0	0	0	0	0	0	0	0		0	0	0	0	0
CPU time		140000 s	73000 s		20000 s	160000 s	200000 s	220000 s	220000 s	10000 s	23000 s	190000 s	160000 s	9100 s	240000 s		11000 s	12000 s	46000 s	230000 s	2300 s
ReachSafety-Recursive 121 tasks		39	90		93	94	98	98	85	58	94	65	66	66	40		87	87	57	90	70
CPU time		49000 s	110000 s		30000 s	33000 s	110000 s	110000 s	78000 s	67000 s	88000 s	72000 s	100000 s	67000 s	42000 s		84000 s	47000 s	71000 s	73000 s	77000 s
ReachSafety-Sequentialized 106 tasks		2	82		84	68	92	91	57	36	78	3	4	2	13		54	59	14	66	63
CPU time		2600 s	82000 s		61000 s	78000 s	95000 s	95000 s	77000 s	29000 s	44000 s	96000 s	95000 s	47000 s	95000 s		44000 s	34000 s	29000 s	70000 s	43000 s
ReachSafety-XCSP 119 tasks		0	118		113	102	107	105	119	102	118	2	73	35	102		78	92	71	118	78
CPU time		1500 s	5100 s		10000 s	88000 s	110000 s	110000 s	6200 s	18000 s	11000 s	110000 s	110000 s	30000 s	110000 s		32000 s	17000 s	33000 s	12000 s	17000 s
ReachSafety-Combinations 671 tasks		89	225		271	282	346	344	165	194	278	179	217	109	79		284	285	93	180	119
CPU time		110000 s	550000 s		510000 s	510000 s	600000 s	600000 s	500000 s	470000 s	370000 s	530000 s	570000 s	550000 s	600000 s		390000 s	390000 s	500000 s	560000 s	530000 s
SoftwareSystems-AWS-C-Common-ReachSafety 343 tasks		4	24		18	26	26	25	5	21	26	7	3	7	6		20	19	0	18	0
CPU time		22000 s	190000 s		830 s	55000 s	300000 s	290000 s	98000 s	29000 s	23000 s	250000 s	290000 s	2900 s	300000 s		25000 s	6300 s	24 s	260000 s	7700 s
SoftwareSystems-BusyBox-MemSafety 76 tasks		0	11		12	26	25	26	5	22	24	0	0	0	15		18	18	0	10	0
CPU time		140 s	68000 s		2700 s	64000 s	62000 s	68000 s	59000 s	60000 s	66000 s	68000 s	68000 s	130 s	66000 s		35000 s	15000 s	5.2 s	44000 s	1000 s
SoftwareSystems-OpenBSD-MemSafety 5 tasks		0	0		0	0	0	0	0	0	0	0	0	0	0		0	0	0	0	0
CPU time		10 s	27 s		6.1 s	9.4 s	3100 s	3000 s	29 s	2.0 s	1.9 s	4500 s	4500 s	7.8 s	670 s		11 s	8.2 s	0.34 s	220 s	91 s
SoftwareSystems-DeviceDriversLinux64-ReachSafety 289 tasks		51	59		52	59	59	59	6	25	58	54	53	0	16		43	56	0	58	0
CPU time		190000 s	38000 s		70000 s	150000 s	260000 s	260000 s	27000 s	60000 s	240000 s	260000 s	250000 s	3300 s	250000 s		240000 s	23000 s	20 s	62000 s	2100 s
SoftwareSystems-SQLite-MemSafety 1 tasks		0	0		0	0	0	0	0	0	0	0	0	0	0		0	0	0	0	0
CPU time		110 s	900 s		33 s	870 s	900 s	900 s	900 s	35 s	880 s	900 s	230 s	11 s	900 s		24 s	58 s	0.067 s	62 s	260 s
Termination-BitVectors 21 tasks		16	21		21	20	21	21	21	11	21	20	21	16	20		18	20	0	21	8
CPU time		15000 s	130 s		530 s	19 s	19000 s	3100 s	130 s	4.1 s	11000 s	16000 s	15000 s	14000 s	18000 s		11000 s	5100 s	1.4 s	430 s	6400 s
Termination-MainControlFlow 392 tasks		245	355		358	350	359	358	348	209	355	351	354	304	284		301	325	21	346	202
CPU time		270000 s	36000 s		100000 s	35000 s	350000 s	78000 s	31000 s	16000 s	300000 s	300000 s	260000 s	310000 s	320000 s		310000 s	210000 s	33000 s	18000 s	210000 s
Termination-MainHeap 251 tasks		70	232		220	216	234	234	211	126	204	185	199	145	70		201	210	0	204	199
CPU time		90000 s	27000 s		35000 s	47000 s	230000 s	180000 s	23000 s	8800 s	190000 s	210000 s	180000 s	83000 s	82000 s		200000 s	70000 s	17 s	10000 s	130000 s
Overall 11042 tasks		2258	4806		5684	5538	7295	7248	4022	4932	5766		3098	2420	2431		5245	4799	2315	4212	4009
CPU time		1700000 s	5800000 s		4600000 s	6400000 s	9000000 s	8700000 s	5100000 s	2700000 s	6100000 s		8400000 s	6100000 s	8300000 s		2800000 s	2100000 s	1600000 s	4500000 s	3500000 s
Participants	Plots	cetfuzz	CoVeriTest	ESBMC-kind	FDSE	Fizzer	FuSeBMC	FuSeBMC-AI	HybridTiger	KLEE	KLEEF	Legion	Legion/SymCC	Owi	PRTest	Rizzer	Symbiotic	TracerX	TracerX-WP	UTestGen	WASP-C

Note on meta-categories: The score is not the sum of scores of the sub-categories (normalization). The run time is the sum of run times of the sub-categories, rounded to two significant digits.

If you did not find what you are looking for, please do not hesitate to contact Dirk Beyer (competition chair).