TOOLympics 2019
1st Competition on Software Testing (Test-Comp 2019)

Results of the Competition

This web page presents the results of Test-Comp 2019 - 1st International Competition on Software Testing.

• The competition definition describes the competition in more detail.
• The benchmarks and rules are also available from the competition web site.

The background color is gold for the winner, silver for the second, and bronze for the third.

Here some brief directions for reading the score-based quantile plots:

• A data point (x, y) tells you that the quantile of cumulative score x can be achieved with a minimum of y tasks.
• The right end of the graph displays the achieved score. The score is the competition's definition of quality. You can read the ranking on the top from right to left.
• More information on what to learn from and how to interpret score-based quantile plots can be found in the SV-COMP '15 report on pages 12 and 13.

Here some brief directions for navigating in the BenchExec-generated tables with the results:

• Please note the header with the menu "Select Columns, Filter Rows, etc.", which loads only after all data are retrieved from the server.
• The tables contain all information that I thought you could be interested in. If there are too many columns, you can get rid of some with "Select Columns". You can also investigate special classes of rows using "Filter Rows".
• If you want to see more data, including the time and memory needed for validation, please click on the score value in the table below (works only for base categories) to browse the most detailed tables, which also contain links to obtain the test-suites that the test-generator generated.
• In all results tables, the sources of the verification task is available via click on "prog". The log files are available via click on the status (result).
• If you want to investigate even more, you can download the raw data that I obtained with BenchExec and from which I compiled all the tables and aggregated data displays. For the most detailed tables, which end on ".xml.bz2.table.html", you can obtain the raw data by accessing the URL with the ".table.html" removed.
• More on table generation is described here.
• Column "score" shows the (not normalized) score assigned to the result of the task.
• Column "test-suite" contains a link to the test-suite archive for download/display.
• Column "inspect test-suite" opens a new page that displays information about the test-suite an a list of test-suites that other test-generators produced for the same program.
• Column "cpu" shows the consumed CPU time.
• Column "wall" shows the elapsed wall time.
• Column "energy" shows the energy consumed by the CPU.
• Column "mem" shows the peak memory usage.
• Column "blkio-w" shows the accumulated amount of information written to block devices.
• Column "blkio-r" shows the accumulated amount of information read from block devices.
• More information on the results are available in the FAQ.

Ranking by Category (with Score-Based Quantile Plots)

Cover-Error 1. VeriFuzz 2. KLEE 3. CoVeriTest	Cover-Branches 1. VeriFuzz 2. KLEE 3. CoVeriTest	Overall 1. VeriFuzz 2. KLEE 3. CoVeriTest

Table of All Results

In every table cell for competition results, we list the points in the first row and the CPU time (rounded to two significant digits) for successful runs in the second row.

The entry '–' means that the competition candidate opted-out in the category.
The definition of the scoring schema and the categories is given on the respective Test-Comp web pages.

Hide base categories

Tools	Plots	CoVeriTest	CPA/Tiger-MGP	ESBMC-bkind 6.1	ESBMC-falsif 6.1	FairFuzz	KLEE	PRTest	Symbiotic	VeriFuzz
Representing Jury Member		Marie-Christine Jakobs	Sebastian Ruland	Rafael Menezes	Mikhail Gadelha	Caroline Lemieux	Cristian Cadar	Thomas Lemberger	Martina Vitovská	Raveendra Kumar Medicherla
Affiliation		LMU Munich, Germany	TU Darmstadt, Germany	Federal University of Amazonas	University of Southampton	University of California, Berkeley	Imperial College London	LMU Munich, Germany	Masaryk University, Czechia	TCS, India
Cover-Error 636 tasks		397	361	237	247	365	499	193	365	595
CPU time		31000 s	17000 s	40000 s	22000 s	150000 s	18000 s	3900 s	25000 s	53000 s
coverage-error-call.ReachSafety-Arrays 22 tasks		1	1	6	6	16	18	5	18	20
CPU time		23 s	3.3 s	190 s	190 s	9800 s	45 s	1.3 s	250 s	510 s
coverage-error-call.ReachSafety-BitVectors 10 tasks		7	7	6	6	8	9	5	5	10
CPU time		150 s	59 s	1.6 s	1.4 s	2100 s	2.0 s	9.0 s	6.2 s	880 s
coverage-error-call.ReachSafety-ControlFlow 8 tasks		8	8			3	8			7
CPU time		58 s	54 s			1300 s	1.5 s			430 s
coverage-error-call.ReachSafety-ECA 412 tasks		195	108	251	254	132	274		255	376
CPU time		23000 s	12000 s	40000 s	22000 s	120000 s	14000 s		23000 s	45000 s
coverage-error-call.ReachSafety-Floats 32 tasks		25	23	6	6	11		24	1	30
CPU time		2500 s	1700 s	.97 s	.95 s	4700 s		1800 s	.54 s	1400 s
coverage-error-call.ReachSafety-Heap 9 tasks		9	9	1	1	9	9	5	8	8
CPU time		130 s	48 s	.10 s	.10 s	140 s	1.4 s	2000 s	5.7 s	35 s
coverage-error-call.ReachSafety-Loops 27 tasks		22	21	17	19	18	24	17	23	26
CPU time		620 s	150 s	18 s	12 s	6100 s	43 s	180 s	35 s	150 s
coverage-error-call.ReachSafety-Recursive 16 tasks		3		15	16	15	16	1	16	15
CPU time		11 s		19 s	770 s	3900 s	34 s	.23 s	8.8 s	1400 s
coverage-error-call.ReachSafety-Sequentialized 100 tasks		61	61				79		46	100
CPU time		4300 s	3000 s				3200 s		1500 s	3200 s
Cover-Branches 1720 tasks		1153	966	0	0	874	1226	476	907	1238
CPU time		1200000 s	1200000 s	48 s	49 s	840000 s	1100000 s	1400000 s	990000 s	1500000 s
coverage-branches.ReachSafety-Arrays 169 tasks		85	76	0	0	124	120	42	116	130
CPU time		140000 s	110000 s	4.6 s	4.9 s	57000 s	34000 s	90000 s	69000 s	150000 s
coverage-branches.ReachSafety-BitVectors 32 tasks		24	22	0	0	21	24	13	21	25
CPU time		9400 s	15000 s	.89 s	.89 s	9400 s	10000 s	23000 s	12000 s	29000 s
coverage-branches.ReachSafety-ControlFlow 19 tasks		14	14	0	0	6	14	1	12	14
CPU time		990 s	1600 s	.49 s	.58 s	5100 s	920 s	17000 s	31 s	17000 s
coverage-branches.ReachSafety-ECA 1046 tasks		360	164	0	0	466	721	86	0	762
CPU time		890000 s	940000 s	29 s	30 s	670000 s	940000 s	940000 s	790000 s	930000 s
coverage-branches.ReachSafety-Floats 212 tasks		104	99	0	0	83	54	84	50	90
CPU time		79000 s	41000 s	5.8 s	5.9 s	60000 s	20000 s	180000 s	19000 s	190000 s
coverage-branches.ReachSafety-Heap 25 tasks		19	19	0	0	16	19	13	18	18
CPU time		2300 s	1000 s	.72 s	.69 s	480 s	900 s	19000 s	11 s	22000 s
coverage-branches.ReachSafety-Loops 117 tasks		92	81	0	0	73	93	60	78	84
CPU time		57000 s	40000 s	3.2 s	3.4 s	31000 s	59000 s	79000 s	55000 s	100000 s
coverage-branches.ReachSafety-Recursive 38 tasks		29	12	0	0	30	33	8	25	31
CPU time		8600 s	110 s	1.0 s	1.1 s	10000 s	31000 s	9000 s	23000 s	34000 s
coverage-branches.ReachSafety-Sequentialized 62 tasks		55	49	0	0	0	49	5	28	49
CPU time		49000 s	48000 s	1.8 s	1.8 s	200 s	26000 s	56000 s	18000 s	55000 s
Overall 2356 tasks		1524	1331	438	457	1275	1764	683	1298	1951
CPU time		1300000 s	1200000 s	40000 s	23000 s	990000 s	1100000 s	1400000 s	1000000 s	1600000 s
Tools	Plots	CoVeriTest	CPA/Tiger-MGP	ESBMC-bkind 6.1	ESBMC-falsif 6.1	FairFuzz	KLEE	PRTest	Symbiotic	VeriFuzz

Note on meta-categories: The score is not the sum of scores of the sub-categories (normalization). The run time is the sum of run times of the sub-categories, rounded to two significant digits.

If you did not find what you are looking for, please do not hesitate to contact Dirk Beyer (competition chair).