| Benchmark | CPA/CoVeriTest | CPA/Tiger-MGP | ESBMC-bkind | ESBMC-falsif | FairFuzz | KLEE | PRtest | Symbiotic | VeriFuzz | ||||||||||||||||||||||||||||||||||||
| Tool | CPAchecker 1.8-svn 30375 | CPAchecker 1.8-svn 30541M | ESBMC version 6.0.0 64-bit x86_64 linux | FairFuzz TC-0.0.2 | KLEE 2.0.0-pre-test-comp | tbf v0.3.0-testcomp19 | symbiotic 6.0.3-dev-fd3f777b | VeriFuzz 1.0.1 | |||||||||||||||||||||||||||||||||||||
| Limits | timelimit: 900 s, memlimit: 15000 MB, CPU core limit: 8 | ||||||||||||||||||||||||||||||||||||||||||||
| Host | apollon* | ||||||||||||||||||||||||||||||||||||||||||||
| OS | Linux 4.15.0-45-generic | ||||||||||||||||||||||||||||||||||||||||||||
| System | CPU: Intel Xeon E3-1230 v5 @ 3.40 GHz, cores: 8, frequency: 3.8 GHz, Turbo Boost: disabled; RAM: 33546 MB | ||||||||||||||||||||||||||||||||||||||||||||
| Date of execution | 2019-02-05 21:05:56 CET | 2019-02-05 21:13:54 CET | 2019-02-05 21:13:25 CET | 2019-02-05 21:11:17 CET | 2019-02-08 09:39:09 CET | 2019-02-05 21:09:47 CET | 2019-02-06 06:02:44 CET | 2019-02-06 07:07:32 CET | 2019-02-06 07:17:17 CET | ||||||||||||||||||||||||||||||||||||
| Run set | coveritest.test-comp19_prop-coverage-error-call.ReachSafety-Loops | cpa-tiger.test-comp19_prop-coverage-error-call.ReachSafety-Loops | esbmc-kind.test-comp19_prop-coverage-error-call.ReachSafety-Loops | esbmc-falsi.test-comp19_prop-coverage-error-call.ReachSafety-Loops | fairfuzz.test-comp19_prop-coverage-error-call.ReachSafety-Loops | klee.test-comp19_prop-coverage-error-call.ReachSafety-Loops | prtest.test-comp19_prop-coverage-error-call.ReachSafety-Loops | symbiotic.test-comp19_prop-coverage-error-call.ReachSafety-Loops | verifuzz.test-comp19_prop-coverage-error-call.ReachSafety-Loops | ||||||||||||||||||||||||||||||||||||
| Options | -benchmark -heap 10000M -testcomp19 | -benchmark -heap 10000M -tigertestcomp19 | -s kinduction | -s falsi | --stats -i random --write-xml --svcomp-nondets | --test-comp --verifier klee-testcomp | --testcomp | ||||||||||||||||||||||||||||||||||||||
| ../sv-benchmarks/c/ | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) |
| loops/array-2.yml | done | 1 | 23 | 1600 | 280 | unknown | 1 | 3.0 | 270 | 27 | false(unreach-call) | 1 | .092 | 26 | 1.2 | false(unreach-call) | 1 | .11 | 26 | .99 | false(unreach-call) | 1 | 17 | 16 | 61 | false(unreach-call) | 1 | .12 | 21 | 1.3 | unknown | 1 | .27 | 31 | 3.5 | done | 1 | .51 | 19 | 6.2 | done | 1 | 3.5 | 150 | 33 |
| loops/compact.yml | timeout | 900 | 2900 | 6200 | timeout | 900 | 2500 | 6000 | timeout | 900 | 12000 | 9700 | timeout | 900 | 11000 | 11000 | timeout | 1 | 900 | 63 | 12000 | timeout | 900 | 2800 | 11000 | timeout | 900 | 35 | 12000 | timeout (verification) | 900 | 320 | 10000 | done | 1 | 4.0 | 150 | 41 | |||||||
| loops/count_up_down-2.yml | done | 1 | 3.3 | 260 | 30 | unknown | 1 | 3.1 | 270 | 30 | false(unreach-call) | 1 | .085 | 26 | 1.1 | false(unreach-call) | 1 | .090 | 26 | .95 | couldn't run: all seeds time out or crash | .13 | 10 | 1.5 | false(unreach-call) | 1 | .12 | 21 | 1.2 | unknown | 1 | 7.6 | 30 | 110 | done | 1 | .45 | 16 | 5.7 | done | 1 | 5.4 | 150 | 54 | |
| loops/eureka_01-1.yml | timeout | 900 | 5100 | 13000 | unknown | 5.6 | 300 | 62 | false(unreach-call) | 1 | .18 | 27 | 2.2 | false(unreach-call) | 1 | .18 | 27 | 2.0 | false(unreach-call) | 1 | 60 | 19 | 170 | false(unreach-call) | 1 | .13 | 21 | 1.6 | timeout | 900 | 31 | 13000 | done | 1 | 1.0 | 19 | 16 | done | 1 | 4.4 | 150 | 51 | |||
| loops/for_bounded_loop1.yml | done | 1 | 3.9 | 270 | 40 | unknown | 1 | 3.1 | 270 | 31 | false(unreach-call) | 1 | .12 | 26 | 1.2 | false(unreach-call) | 1 | .12 | 26 | .85 | false(unreach-call) | 1 | 110 | 19 | 340 | false(unreach-call) | 1 | .14 | 21 | 1.2 | unknown | 1 | 140 | 31 | 2200 | done | 1 | .58 | 19 | 6.8 | done | 1 | 3.4 | 150 | 29 |
| loops/insertion_sort-1.yml | done | 1 | 320 | 3900 | 4000 | unknown | 3.1 | 270 | 31 | false(unreach-call) | .81 | 28 | 9.7 | false(unreach-call) | 1 | .63 | 28 | 7.7 | false(unreach-call) | 1 | 35 | 16 | 440 | false(valid-deref) | .12 | 21 | 1.4 | error (0) | .26 | 31 | 4.0 | timeout (verification) | 900 | 1500 | 11000 | done | 1 | 3.6 | 150 | 41 | |||||
| loops/invert_string-1.yml | timeout | 900 | 5300 | 13000 | unknown | 3.1 | 270 | 30 | false(unreach-call) | 1 | .099 | 26 | 1.0 | false(unreach-call) | 1 | .10 | 26 | 1.0 | couldn't run: all seeds time out or crash | 2.2 | 11 | 28 | false(unreach-call) | 1 | .13 | 21 | 1.4 | error (0) | .25 | 31 | 3.0 | done | 1 | .90 | 20 | 11 | done | 1 | 3.4 | 150 | 34 | ||||
| loops/linear_search.yml | timeout | 900 | 1500 | 11000 | unknown | 3.1 | 270 | 30 | done | .13 | 27 | 2.2 | false(unreach-call) | .32 | 27 | 4.4 | false(unreach-call) | 1 | 30 | 19 | 130 | done | .17 | 21 | 1.6 | error (0) | .19 | 31 | 2.9 | done | 1 | 3.8 | 21 | 48 | done | 1 | 3.7 | 150 | 41 | ||||||
| loops/matrix-2.yml | done | 1 | 25 | 880 | 200 | unknown | 1 | 3.3 | 270 | 30 | false(unreach-call) | 1 | .15 | 26 | 1.6 | false(unreach-call) | 1 | .14 | 26 | 1.7 | timeout | 1 | 900 | 180 | 13000 | false(valid-deref) | 1 | 39 | 120 | 590 | error (0) | .26 | 31 | 4.3 | done | 1 | 12 | 30 | 170 | done | 1 | 12 | 150 | 130 | |
| loops/nec20.yml | done | 1 | 3.6 | 270 | 35 | unknown | 1 | 3.0 | 270 | 26 | false(unreach-call) | .088 | 26 | .90 | false(unreach-call) | .089 | 26 | .93 | false(unreach-call) | 1 | 8.7 | 16 | 30 | false(unreach-call) | 1 | .14 | 21 | 1.2 | unknown | 1 | .51 | 31 | 6.1 | done | 1 | .56 | 19 | 7.1 | done | 1 | 3.8 | 150 | 37 | ||
| loops/string-2.yml | done | 1 | 4.6 | 280 | 44 | unknown | 1 | 4.3 | 290 | 41 | error (1) | .098 | 26 | 1.0 | error (1) | .085 | 26 | .96 | false(unreach-call) | 1 | 45 | 16 | 220 | false(unreach-call) | 1 | .15 | 21 | .84 | timeout | 900 | 30 | 13000 | done | 1 | .78 | 19 | 9.1 | done | 1 | 4.0 | 150 | 41 | |||
| loops/sum01-1.yml | done | 1 | 24 | 850 | 190 | unknown | 1 | 4.7 | 280 | 37 | false(unreach-call) | 1 | .15 | 26 | 1.8 | false(unreach-call) | 1 | .13 | 26 | 1.5 | false(unreach-call) | 1 | 22 | 18 | 53 | false(unreach-call) | 1 | .14 | 21 | 1.1 | unknown | 1 | 11 | 31 | 140 | done | 1 | .72 | 20 | 9.5 | done | 1 | 6.7 | 160 | 69 |
| loops/sum01_bug02.yml | done | 1 | 24 | 860 | 190 | unknown | 1 | 4.0 | 280 | 34 | done | .11 | 26 | 1.0 | false(unreach-call) | 1 | .11 | 26 | 1.1 | false(unreach-call) | 1 | 560 | 11 | 7700 | false(unreach-call) | 1 | .13 | 21 | 1.2 | unknown | 1 | 9.6 | 31 | 130 | done | 1 | .66 | 20 | 8.9 | done | 1 | 8.9 | 150 | 98 | |
| loops/sum01_bug02_sum01_bug02_base.case.yml | done | 1 | 9.3 | 450 | 83 | unknown | 1 | 3.5 | 280 | 29 | false(unreach-call) | 1 | .14 | 26 | 1.9 | false(unreach-call) | 1 | .096 | 26 | 1.1 | false(unreach-call) | 1 | 770 | 10 | 11000 | false(unreach-call) | 1 | .12 | 21 | 1.3 | unknown | 1 | 11 | 31 | 180 | done | 1 | .61 | 20 | 8.4 | done | 1 | 9.4 | 150 | 79 |
| loops/sum03-1.yml | done | 1 | 4.9 | 280 | 40 | unknown | 1 | 4.6 | 290 | 39 | false(unreach-call) | .13 | 26 | 1.5 | false(unreach-call) | .089 | 26 | 1.0 | couldn't run: all seeds time out or crash | .16 | 10 | 1.5 | false(unreach-call) | 1 | .13 | 21 | 1.1 | unknown | 1 | .27 | 31 | 3.1 | done | 1 | .51 | 16 | 6.1 | done | 1 | 3.4 | 150 | 34 | |||
| loops/sum_array-1.yml | done | 1 | 23 | 820 | 220 | unknown | 1 | 3.4 | 270 | 34 | false(unreach-call) | 1 | .11 | 26 | 1.3 | false(unreach-call) | 1 | .11 | 26 | 1.4 | false(unreach-call) | 1 | 710 | 150 | 9300 | false(valid-deref) | 1 | .14 | 21 | 1.5 | error (0) | .26 | 31 | 3.0 | done | 1 | .75 | 19 | 9.3 | done | 1 | 10 | 150 | 130 | |
| loops/terminator_01.yml | done | 1 | 3.4 | 260 | 29 | unknown | 1 | 2.8 | 260 | 30 | false(unreach-call) | 1 | .087 | 26 | .95 | false(unreach-call) | 1 | .13 | 26 | .84 | couldn't run: all seeds time out or crash | .15 | 11 | 1.3 | false(unreach-call) | 1 | .13 | 22 | 1.3 | unknown | 1 | 3.7 | 31 | 53 | done | 1 | .44 | 17 | 6.5 | done | 1 | 3.4 | 150 | 35 | |
| loops/terminator_02-1.yml | done | 1 | 3.4 | 260 | 31 | unknown | 1 | 2.9 | 260 | 24 | false(unreach-call) | .085 | 26 | .66 | false(unreach-call) | .11 | 26 | .68 | couldn't run: all seeds time out or crash | .13 | 10 | 1.6 | false(unreach-call) | 1 | .14 | 21 | 1.0 | unknown | 1 | .29 | 31 | 3.3 | done | 1 | .58 | 19 | 9.4 | done | 1 | 3.4 | 150 | 35 | |||
| loops/terminator_03-1.yml | done | 1 | 23 | 1400 | 300 | unknown | 1 | 2.9 | 260 | 28 | false(unreach-call) | 1 | .083 | 26 | 1.0 | false(unreach-call) | 1 | .081 | 26 | .85 | false(unreach-call) | 1 | 10 | 16 | 100 | false(unreach-call) | 1 | .13 | 21 | 1.3 | unknown | 1 | .24 | 31 | 3.5 | done | .49 | 19 | 7.2 | done | 1 | 3.3 | 150 | 31 | |
| loops/trex01-1.yml | done | 1 | 3.5 | 260 | 36 | unknown | 1 | 3.1 | 270 | 28 | false(unreach-call) | .086 | 26 | .95 | false(unreach-call) | .090 | 26 | .83 | couldn't run: all seeds time out or crash | 1.1 | 18 | 16 | false(unreach-call) | 1 | .15 | 21 | 1.1 | unknown | 1 | .28 | 31 | 2.9 | done | .64 | 19 | 8.7 | done | 1 | 3.6 | 150 | 40 | ||||
| loops/trex02-2.yml | done | 1 | 3.4 | 260 | 36 | unknown | 1 | 3.1 | 270 | 26 | false(unreach-call) | 1 | .084 | 26 | 1.1 | false(unreach-call) | 1 | .088 | 26 | .99 | timeout | 1 | 900 | 41 | 11000 | false(unreach-call) | 1 | .16 | 21 | .98 | unknown | 1 | .24 | 31 | 2.9 | done | 1 | .56 | 19 | 6.4 | done | 1 | 24 | 150 | 290 |
| loops/trex03-1.yml | done | 1 | 22 | 1500 | 240 | unknown | 1 | 2.9 | 270 | 27 | false(unreach-call) | .084 | 26 | .92 | false(unreach-call) | .083 | 26 | .92 | timeout | 1 | 900 | 69 | 10000 | false(unreach-call) | 1 | .13 | 21 | 1.1 | timeout | 900 | 35 | 14000 | done | 1 | .66 | 19 | 7.6 | error | 12 | 150 | 160 | ||||
| loops/vogal-2.yml | done | 1 | 72 | 900 | 700 | unknown | 1 | 48 | 500 | 590 | false(unreach-call) | 1 | .41 | 34 | 5.7 | false(unreach-call) | 1 | .31 | 34 | 3.8 | false(unreach-call) | 1 | 130 | 19 | 390 | false(unreach-call) | 1 | .15 | 21 | 1.1 | unknown | 1 | .25 | 30 | 3.3 | done | 1 | .78 | 19 | 11 | done | 1 | 3.8 | 150 | 40 |
| loop-acceleration/array_3-2.yml | timeout | 900 | 5700 | 12000 | timeout | 900 | 6500 | 13000 | false(unreach-call) | 1 | 16 | 370 | 240 | false(unreach-call) | 1 | 9.3 | 260 | 140 | done | 510 | 11 | 3200 | false(unreach-call) | 1 | .93 | 21 | 14 | unknown | 1 | .26 | 31 | 3.0 | done | 1 | 6.5 | 21 | 87 | done | 1 | 3.4 | 150 | 38 | |||
| loop-acceleration/diamond_1-2.yml | done | 1 | 10 | 500 | 90 | unknown | 1 | 35 | 560 | 460 | false(unreach-call) | 1 | .45 | 26 | 5.7 | false(unreach-call) | 1 | .28 | 26 | 3.3 | couldn't run: all seeds time out or crash | .14 | 11 | 1.8 | false(unreach-call) | 1 | .13 | 21 | 1.2 | unknown | 1 | .26 | 31 | 4.1 | done | 1 | .49 | 19 | 6.6 | done | 1 | 3.5 | 150 | 34 | |
| loop-acceleration/multivar_1-2.yml | done | 1 | 3.4 | 260 | 34 | unknown | 1 | 2.9 | 270 | 28 | false(unreach-call) | 1 | .083 | 26 | .83 | false(unreach-call) | 1 | .089 | 26 | 1.0 | couldn't run: all seeds time out or crash | .14 | 11 | 1.5 | false(unreach-call) | 1 | .12 | 21 | 1.1 | unknown | 1 | .25 | 31 | 3.2 | done | 1 | .46 | 16 | 5.7 | done | 1 | 3.4 | 150 | 34 | |
| loop-invgen/id_trans.yml | done | 1 | 3.8 | 270 | 40 | unknown | 1 | 3.0 | 270 | 31 | false(unreach-call) | 1 | .10 | 26 | 1.0 | false(unreach-call) | 1 | .095 | 26 | 1.1 | false(unreach-call) | 1 | 9.4 | 16 | 44 | false(unreach-call) | 1 | .13 | 21 | 1.2 | timeout | 900 | 31 | 13000 | done | 1 | .60 | 24 | 7.5 | done | 1 | 4.1 | 150 | 46 | |
| ../sv-benchmarks/c/ | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) | status | score | cpu (s) | mem (MB) | energy (J) |
| total | 27 | 22 | 5100 | 37000 | 61000 | 27 | 21 | 2000 | 16000 | 21000 | 27 | 17 | 920 | 13000 | 10000 | 27 | 19 | 910 | 12000 | 11000 | 27 | 18 | 6600 | 820 | 80000 | 27 | 24 | 940 | 3400 | 11000 | 27 | 17 | 4700 | 840 | 68000 | 27 | 23 | 1800 | 2300 | 22000 | 27 | 26 | 160 | 4100 | 1700 |
| Run set | coveritest.test-comp19_prop-coverage-error-call.ReachSafety-Loops | cpa-tiger.test-comp19_prop-coverage-error-call.ReachSafety-Loops | esbmc-kind.test-comp19_prop-coverage-error-call.ReachSafety-Loops | esbmc-falsi.test-comp19_prop-coverage-error-call.ReachSafety-Loops | fairfuzz.test-comp19_prop-coverage-error-call.ReachSafety-Loops | klee.test-comp19_prop-coverage-error-call.ReachSafety-Loops | prtest.test-comp19_prop-coverage-error-call.ReachSafety-Loops | symbiotic.test-comp19_prop-coverage-error-call.ReachSafety-Loops | verifuzz.test-comp19_prop-coverage-error-call.ReachSafety-Loops | ||||||||||||||||||||||||||||||||||||