FASE 2025
7th Competition on Software Testing (Test-Comp 2025)

Results of the Competition

This web page presents the results of Test-Comp 2025 - 7th International Competition on Software Testing.

• Competition Report for Test-Comp 2025
• The benchmarks and rules are also available from the competition web site.

The background color is gold for the winner, silver for the second, and bronze for the third.

Here some brief directions for reading the score-based quantile plots:

• A data point (x, y) tells you that the quantile of cumulative score x can be achieved with a minimum of y tasks.
• The right end of the graph displays the achieved score. The score is the competition's definition of quality. You can read the ranking on the top from right to left.
• More information on what to learn from and how to interpret score-based quantile plots can be found in the SV-COMP '15 report on pages 12 and 13.

Here some brief directions for navigating in the BenchExec-generated tables with the results:

• Please note the header with the menu "Select Columns, Filter Rows, etc.", which loads only after all data are retrieved from the server.
• The tables contain all information that I thought you could be interested in. If there are too many columns, you can get rid of some with "Select Columns". You can also investigate special classes of rows using "Filter Rows".
• If you want to see more data, including the time and memory needed for validation, please click on the score value in the table below (works only for base categories) to browse the most detailed tables, which also contain links to obtain the test-suites that the test-generator generated.
• In all results tables, the sources of the verification task is available via click on "prog". The log files are available via click on the status (result).
• If you want to investigate even more, you can download the Test-Comp 2025 results archive, which contains all data that I obtained with BenchExec and from which I compiled all the tables and aggregated data displays. For the most detailed tables, which end on ".xml.bz2.table.html", you can obtain the raw data by accessing the URL with the ".table.html" removed.
• More on table generation is described here.
• Column "score" shows the (not normalized) score assigned to the result of the task.
• Column "test-suite" contains a link to the test-suite archive for download/display.
• Column "inspect test-suite" opens a new page that displays information about the test-suite an a list of test-suites that other test-generators produced for the same program.
• Column "cpu" shows the consumed CPU time.
• Column "wall" shows the elapsed wall time.
• Column "energy" shows the energy consumed by the CPU.
• Column "mem" shows the peak memory usage.
• Column "blkio-w" shows the accumulated amount of information written to block devices.
• Column "blkio-r" shows the accumulated amount of information read from block devices.
• More information on the results are available in the FAQ.

Ranking by Category (with Score-Based Quantile Plots)

Cover-Error 1. FuSeBMC 2. KLEEF 3. Symbiotic	Cover-Branches 1. KLEEF 2. FuSeBMC 3. FDSE	Overall 1. FuSeBMC 2. KLEEF 3. Fizzer

Table of All Results

In every table cell for competition results, we list the points in the first row and the CPU time (rounded to two significant digits) for successful runs in the second row.

The entry '–' means that the competition candidate opted-out in the category.
The definition of the scoring schema and the categories is given on the respective SV-COMP web pages.

Hide base categories

Participants	Plots	cetfuzz	CoVeriTest	ESBMC-incr	ESBMC-kind	FDSE	Fizzer	FuSeBMC	FuSeBMC-AI	HybridTiger	KLEE	KLEEF	Owi	PRTest	Rizzer	Sikraken	Symbiotic	TracerX	TracerX-WP	UTestGen	WASP-C
Representing Jury Member		Hors Concours	Marie-Christine Jakobs	Chenfeng Wei	Chenfeng Wei	Zhenbang Chen	Marek Trtík	Kaled Alshmrany	Hors Concours	Hors Concours	Hors Concours	Aleksandr Misonizhnik	Hors Concours	Thomas Lemberger	Hors Concours	Christophe Meudec	Martin Jonáš	Joxan Jaffar	Joxan Jaffar	Max Barth	Hors Concours
Affiliation		--, --	LMU Munich, Germany	University of Manchester, UK	University of Manchester, UK	National University of Defense Technology, China	Masaryk University, Brno, Czechia	University of Manchester / Institute of Public Administration, UK / Saudi Arabia	--, --	--, --	--, --	Independent Researcher, Neutral	--, --	LMU Munich, Germany	--, --	South East Technological University, Ireland	Masaryk University, Brno, Czechia	National University of Singapore, Singapore	National University of Singapore, Singapore	LMU Munich, Germany	--, --
Cover-Error 1215 valid tasks		323	552	679	680	729	736	994	853	438	804	969	281	211	608		743	390	349	439	554
CPU time		98000 s	130000 s	30000 s	31000 s	540000 s	410000 s	270000 s	160000 s	260000 s	26000 s	34000 s	3300 s	230000 s	290000 s		20000 s	10000 s	12000 s	59000 s	9300 s
ReachSafety-Arrays 100 valid tasks		6	73	70	70	96	80	99	84	69	89	89	45	37	82		74	56	52	73	91
CPU time		2400 s	8100 s	130 s	130 s	85000 s	61000 s	21000 s	7600 s	41000 s	100 s	1500 s	53 s	5000 s	57000 s		660 s	130 s	62 s	1900 s	1200 s
ReachSafety-BitVectors 9 valid tasks		5	7	8	8	8	8	9	9	5	8	9	5	5	7		8	3	5	3	7
CPU time		810 s	230 s	15 s	30 s	6200 s	4700 s	880 s	930 s	1400 s	4.5 s	3.3 s	21 s	4500 s	3500 s		48 s	0.92 s	8.2 s	210 s	510 s
ReachSafety-ControlFlow 5 valid tasks		3	2	0	0	4	5	5	2	1	5	5	1	0	2		5	3	3	2	5
CPU time		2400 s	1700 s	0 s	0 s	3500 s	4200 s	430 s	52 s	900 s	2.8 s	4.2 s	3.8 s	0 s	1000 s		120 s	2.0 s	2.3 s	840 s	110 s
ReachSafety-ECA 18 valid tasks		7	3	8	8	10	7	14	13	1	14	13	2	0	7		15	5	2	4	3
CPU time		5800 s	350 s	300 s	500 s	9000 s	6300 s	3000 s	2900 s	900 s	510 s	1200 s	25 s	0 s	6100 s		790 s	120 s	970 s	1100 s	1100 s
ReachSafety-Floats 33 valid tasks		5	25	30	30	28	32	33	33	22	8	28	3	3	18		0	0	0	19	23
CPU time		2400 s	3200 s	950 s	950 s	21000 s	9700 s	4800 s	3600 s	4100 s	2.9 s	600 s	6.4 s	2700 s	100 s		0 s	0 s	0 s	2600 s	2500 s
ReachSafety-Fuzzle 15 valid tasks		0	1	1	1	0	0	15	15	1	1	15	0	1	0		1	1	1	14	0
CPU time		0 s	56 s	14 s	11 s	0 s	0 s	2100 s	1500 s	900 s	4.0 s	620 s	0 s	900 s	0 s		22 s	5.3 s	3.1 s	3000 s	0 s
ReachSafety-Heap 53 valid tasks		21	48	45	46	47	51	51	51	42	50	51	38	12	50		51	38	28	42	40
CPU time		8100 s	16000 s	140 s	1000 s	39000 s	32000 s	1900 s	180 s	11000 s	160 s	910 s	52 s	11000 s	26000 s		240 s	1500 s	3700 s	11000 s	370 s
ReachSafety-Loops 123 valid tasks		64	71	85	85	114	120	121	117	50	78	85	67	82	110		77	57	69	81	106
CPU time		5700 s	38000 s	1900 s	2200 s	94000 s	75000 s	30000 s	38000 s	31000 s	3300 s	2500 s	3000 s	73000 s	29000 s		2400 s	120 s	1400 s	4900 s	630 s
ReachSafety-ProductLines 169 valid tasks		169	163	169	169	168	166	169	169	48	169	169	50	92	166		159	124	120	35	5
CPU time		63000 s	37000 s	240 s	210 s	150000 s	120000 s	990 s	1200 s	43000 s	86 s	110 s	110 s	83000 s	110000 s		3000 s	240 s	220 s	28000 s	35 s
ReachSafety-Recursive 21 valid tasks		11	7	18	18	18	17	20	20	5	17	18	15	3	15		18	11	2	14	14
CPU time		8100 s	4000 s	40 s	36 s	16000 s	8800 s	890 s	1800 s	690 s	3.9 s	4.8 s	37 s	2700 s	1500 s		100 s	25 s	18 s	750 s	62 s
ReachSafety-Sequentialized 106 valid tasks		0	61	91	91	63	90	102	101	91	86	97	0	0	49		79	41	50	2	48
CPU time		0 s	5400 s	440 s	440 s	53000 s	80000 s	12000 s	12000 s	80000 s	3600 s	4800 s	0 s	0 s	43000 s		4500 s	1500 s	990 s	1000 s	860 s
ReachSafety-XCSP 59 valid tasks		0	50	55	55	10	0	52	52	52	37	53	1	0	0		46	21	16	13	53
CPU time		0 s	2300 s	2400 s	2400 s	8800 s	0 s	1200 s	820 s	23000 s	1900 s	2900 s	2.1 s	0 s	0 s		7300 s	4800 s	3500 s	2100 s	1400 s
ReachSafety-Hardware 497 valid tasks		26	86	368	367	61	35	288	285	38	323	362	0	57	41		49	58	51	12	37
CPU time		48 s	11000 s	23000 s	23000 s	55000 s	9100 s	190000 s	89000 s	23000 s	16000 s	19000 s	0 s	51000 s	13000 s		620 s	1700 s	1700 s	1800 s	470 s
SoftwareSystems-BusyBox-MemSafety 4 valid tasks		0	0	0	0	0	0	0	0	0	1	1	0	0	0		1	0	0	0	0
CPU time		0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	1.7 s	1.5 s	0 s	0 s	0 s		13 s	0 s	0 s	0 s	0 s
SoftwareSystems-OpenBSD-MemSafety 1 valid tasks		0	0	0	0	1	1	1	0	0	1	1	0	0	1		1	0	0	0	0
CPU time		0 s	0 s	0 s	0 s	870 s	880 s	1.3 s	0 s	0 s	0.40 s	1.0 s	0 s	0 s	2.4 s		1.1 s	0 s	0 s	0 s	0 s
SoftwareSystems-DeviceDriversLinux64-ReachSafety 2 valid tasks		0	0	0	0	0	0	0	0	0	0	0	0	0	0		0	0	0	0	0
CPU time		0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s	0 s		0 s	0 s	0 s	0 s	0 s
Cover-Branches 10011 valid tasks		2524	4959	4380	4323	5468	5429	5656	4077	3866	3065	5734	2462	3191		2469	4207	3327	3275	4393	2740
CPU time		1800000 s	6000000 s	1500000 s	980000 s	7900000 s	7300000 s	8900000 s	3600000 s	4800000 s	2700000 s	5400000 s	6200000 s	8200000 s		7300000 s	2800000 s	2000000 s	2300000 s	5900000 s	3600000 s
ReachSafety-Arrays 468 valid tasks		97	317	290	274	382	373	389	310	306	152	378	153	176		47	303	226	197	305	298
CPU time		150000 s	330000 s	410 s	140 s	320000 s	340000 s	420000 s	190000 s	320000 s	63000 s	270000 s	140000 s	170000 s		420000 s	360000 s	110000 s	91000 s	330000 s	260000 s
ReachSafety-BitVectors 62 valid tasks		36	49	38	35	49	49	50	38	16	32	47	29	33		7	46	34	36	46	34
CPU time		35000 s	20000 s	120 s	72 s	52000 s	49000 s	56000 s	28000 s	7600 s	22000 s	21000 s	35000 s	55000 s		47000 s	35000 s	24000 s	27000 s	36000 s	30000 s
ReachSafety-ControlFlow 87 valid tasks		46	50	38	31	55	55	58	20	21	36	57	19	14		3	56	36	39	43	45
CPU time		60000 s	47000 s	12000 s	9800 s	73000 s	72000 s	77000 s	25000 s	48000 s	40000 s	35000 s	27000 s	36000 s		34000 s	41000 s	27000 s	32000 s	55000 s	31000 s
ReachSafety-ECA 30 valid tasks		9	7	4	3	10	7	12	6	2	8	12	3	2		10	11	10	4	4	4
CPU time		20000 s	25000 s	17000 s	11000 s	26000 s	18000 s	27000 s	17000 s	24000 s	27000 s	23000 s	24000 s	27000 s		27000 s	13000 s	19000 s	22000 s	21000 s	24000 s
ReachSafety-Floats 792 valid tasks		43	230	434	442	511	512	466	279	449	29	500	32	96		127	127	63	71	118	62
CPU time		110000 s	520000 s	84000 s	72000 s	700000 s	490000 s	710000 s	300000 s	500000 s	16000 s	310000 s	450000 s	710000 s		560000 s	27000 s	8000 s	13000 s	360000 s	48000 s
ReachSafety-Fuzzle 15 valid tasks		0	0	0	0	2	0	4	2	0	0	5	0	3		1	1	1	1	0	0
CPU time		13000 s	14000 s	14000 s	14000 s	13000 s	13000 s	13000 s	5400 s	14000 s	6700 s	12000 s	13000 s	14000 s		14000 s	4400 s	9800 s	12000 s	14000 s	13000 s
ReachSafety-Hardness 3605 valid tasks		26	2080	2168	2273	2584	2653	2557	1128	2085	734	2727	58	645		1385	336	164	276	1688	123
CPU time		16000 s	2400000 s	700000 s	470000 s	3200000 s	3200000 s	3200000 s	1400000 s	1400000 s	750000 s	2700000 s	3000000 s	3200000 s		3000000 s	290000 s	130000 s	210000 s	2500000 s	880000 s
ReachSafety-Hardware 1071 valid tasks		638	880	671	594	957	880	754	317	634	292	878	0	968		75	96	491	703	514	19
CPU time		8300 s	840000 s	9000 s	2000 s	960000 s	930000 s	960000 s	390000 s	860000 s	680000 s	850000 s	940000 s	960000 s		960000 s	220000 s	650000 s	740000 s	830000 s	920000 s
ReachSafety-Heap 288 valid tasks		85	174	152	152	182	184	187	171	133	155	185	125	79		5	171	120	123	167	128
CPU time		100000 s	140000 s	18000 s	7900 s	240000 s	130000 s	260000 s	97000 s	74000 s	93000 s	120000 s	100000 s	160000 s		170000 s	110000 s	34000 s	46000 s	120000 s	82000 s
ReachSafety-Loops 778 valid tasks		280	611	502	505	630	627	630	507	499	419	623	433	387		509	581	400	405	603	462
CPU time		270000 s	480000 s	120000 s	110000 s	660000 s	570000 s	700000 s	290000 s	420000 s	230000 s	240000 s	250000 s	680000 s		660000 s	310000 s	150000 s	170000 s	280000 s	220000 s
ReachSafety-ProductLines 263 valid tasks		66	77	71	69	77	76	77	59	55	74	76	59	48		0	69	53	52	52	1
CPU time		140000 s	53000 s	22000 s	20000 s	230000 s	160000 s	190000 s	96000 s	220000 s	10000 s	5100 s	6700 s	240000 s		180 s	8200 s	10000 s	30000 s	220000 s	1700 s
ReachSafety-Recursive 121 valid tasks		40	90	70	70	96	95	97	84	85	58	97	65	42		44	88	59	64	90	73
CPU time		51000 s	110000 s	24000 s	24000 s	110000 s	73000 s	110000 s	51000 s	77000 s	66000 s	65000 s	66000 s	44000 s		110000 s	82000 s	37000 s	58000 s	72000 s	77000 s
ReachSafety-Sequentialized 107 valid tasks		2	83	78	77	81	88	92	41	58	37	93	2	13		73	54	34	34	64	64
CPU time		2600 s	83000 s	4000 s	2900 s	88000 s	94000 s	96000 s	42000 s	75000 s	28000 s	41000 s	47000 s	96000 s		76000 s	44000 s	36000 s	24000 s	72000 s	43000 s
ReachSafety-XCSP 119 valid tasks		0	116	104	104	115	103	107	104	119	102	118	36	102		119	78	58	57	118	79
CPU time		1300 s	4800 s	14000 s	14000 s	18000 s	110000 s	110000 s	52000 s	5900 s	15000 s	22000 s	29000 s	110000 s		110000 s	30000 s	36000 s	37000 s	11000 s	15000 s
ReachSafety-Combinations 671 valid tasks		89	234	167	243	407	287	403	226	166	197	400	109	79		265	369	256	154	184	152
CPU time		110000 s	540000 s	440000 s	170000 s	600000 s	420000 s	600000 s	240000 s	500000 s	470000 s	450000 s	550000 s	600000 s		550000 s	370000 s	290000 s	370000 s	530000 s	520000 s
SoftwareSystems-AWS-C-Common-ReachSafety 343 valid tasks		4	24	20	21	5	26	26	18	5	21	27	7	6		0	20	19	17	19	0
CPU time		22000 s	190000 s	32000 s	28000 s	240 s	250000 s	300000 s	110000 s	98000 s	28000 s	17000 s	2500 s	300000 s		9.5 s	23000 s	6400 s	1700 s	260000 s	5100 s
SoftwareSystems-BusyBox-MemSafety 75 valid tasks		0	15	0	0	0	24	24	12	6	22	28	0	15		0	17	14	15	12	0
CPU time		120 s	67000 s	21 s	21 s	1900 s	65000 s	62000 s	30000 s	46000 s	51000 s	59000 s	82 s	66000 s		53 s	35000 s	7100 s	18000 s	45000 s	730 s
SoftwareSystems-coreutils 108 valid tasks		0	0	0	0	0	0	0	0	0	0	0	0	0		0	0	0	0	0	0
CPU time		97000 s	2200 s	1600 s	1600 s	76000 s	24000 s	96000 s	48000 s	2700 s	730 s	19000 s	910 s	220 s		34 s	1600 s	390 s	330 s	3700 s	35000 s
SoftwareSystems-OpenBSD-MemSafety 5 valid tasks		0	0	0	0	0	0	0	0	0	0	0	0	0		0	0	0	0	0	0
CPU time		9.5 s	23 s	900 s	900 s	4400 s	2600 s	3100 s	590 s	24 s	2.0 s	4.2 s	4.6 s	650 s		0.069 s	9.9 s	4.4 s	4.8 s	200 s	58 s
SoftwareSystems-DeviceDriversLinux64-ReachSafety 287 valid tasks		50	62	59	60	59	61	61	53	5	26	59	0	16		0	43	54	57	61	0
CPU time		190000 s	40000 s	13000 s	8700 s	260000 s	160000 s	260000 s	100000 s	26000 s	60000 s	52000 s	3000 s	240000 s		2.8 s	230000 s	38000 s	45000 s	56000 s	1600 s
SoftwareSystems-uthash 51 valid tasks		35	27	24	24	35	35	35	29	0	6	35	29	34		0	35	0	0	26	0
CPU time		41000 s	45000 s	240 s	210 s	45000 s	44000 s	46000 s	20000 s	27000 s	35000 s	32000 s	31000 s	46000 s		36000 s	33000 s	4.1 s	4.1 s	46000 s	320 s
SoftwareSystems-SQLite-MemSafety 1 valid tasks		0	0	0	0	0	0	0	0	0	0	0	0	0		0	0	0	0	0	0
CPU time		110 s	900 s	15 s	900 s	900 s	870 s	900 s	0.16 s	900 s	30 s	420 s	9.4 s	900 s		0.032 s	22 s	7.3 s	7.3 s	62 s	180 s
Termination-BitVectors 21 valid tasks		18	21	21	21	21	21	21	20	21	11	20	16	20		18	18	20	20	21	8
CPU time		15000 s	110 s	4.4 s	3.8 s	940 s	9.1 s	19000 s	950 s	130 s	4.1 s	5.6 s	14000 s	18000 s		19000 s	11000 s	8900 s	9600 s	400 s	6200 s
Termination-MainControlFlow 392 valid tasks		254	360	353	352	364	361	363	323	354	214	361	307	293		339	306	320	307	353	209
CPU time		270000 s	34000 s	11000 s	9200 s	94000 s	59000 s	350000 s	34000 s	31000 s	17000 s	37000 s	320000 s	320000 s		350000 s	310000 s	260000 s	270000 s	18000 s	210000 s
Termination-MainHeap 251 valid tasks		78	230	202	201	227	229	232	208	215	131	227	149	78		12	207	207	200	223	204
CPU time		90000 s	26000 s	140 s	97 s	63000 s	63000 s	230000 s	79000 s	23000 s	8600 s	32000 s	83000 s	81000 s		170000 s	200000 s	86000 s	84000 s	9300 s	130000 s
Overall 11226 valid tasks		2906	5333	5591	5565	6435	6446	7763	6228	4193	5434	7692	2677	2764			5793	3667	3447	4492	4094
CPU time		1900000 s	6200000 s	1600000 s	1000000 s	8400000 s	7700000 s	9200000 s	3800000 s	5100000 s	2700000 s	5400000 s	6200000 s	8500000 s			2800000 s	2000000 s	2300000 s	6000000 s	3600000 s
Participants	Plots	cetfuzz	CoVeriTest	ESBMC-incr	ESBMC-kind	FDSE	Fizzer	FuSeBMC	FuSeBMC-AI	HybridTiger	KLEE	KLEEF	Owi	PRTest	Rizzer	Sikraken	Symbiotic	TracerX	TracerX-WP	UTestGen	WASP-C

Note on meta-categories: The score is not the sum of scores of the sub-categories (normalization). The run time is the sum of run times of the sub-categories, rounded to two significant digits.

If you did not find what you are looking for, please do not hesitate to contact Dirk Beyer (competition chair).