Benchmark FairFuzz TBF Test-Suite Validator
Tool FairFuzz TC-0.0.2 Tbf Test-suite Validator v1.2
Limits timelimit: 900 s, memlimit: 15000 MB, CPU core limit: 8 timelimit: 10800 s, memlimit: 7000 MB, CPU core limit: 2
Host apollon*
OS Linux 4.15.0-45-generic
System CPU: Intel Xeon E3-1230 v5 @ 3.40 GHz, cores: 8, frequency: 3.8 GHz, Turbo Boost: disabled; RAM: 33546 MB
Date of execution 2019-02-08 09:39:09 CET 2019-02-08 13:22:04 CET
Run set fairfuzz.test-comp19_prop-coverage-branches.ReachSafety-Loops tbf-testsuite-validator-fairfuzz.test-comp19_prop-coverage-branches.ReachSafety-Loops
Options --verbose --sequence-file cov-seq.txt -r --test-suite ../../results-verified/fairfuzz.2019-02-08_0939.logfiles/${rundefinition_name}.${inputfile_name}.files/test-suite.zip
sv-benchmarks/c/ status score program test-suite TS size inspect test-suite cpu (s) wall (s) mem (MB) energy (J) blkio-w (MB) blkio-r (MB) status program test-suite TS size inspect test-suite cov branch cov (%) line cov (%) cpu (s) wall (s) mem (MB) energy blkio-w (MB) blkio-r (MB)
loops/array-1.yml .667 19    18    16 91   .21 0     66.7 93.3 .11  .11  16   .078 0  
loops/array-2.yml 1.0   17    15    17 51   .21 0     100   100   .12  .14  16   .094 0  
loops/bubble_sort-1.yml .0   14    13    20 55   .25 0     0   0   .060 .061 9.0 0     0  
loops/compact.yml 1.0   900    910    54 12000   .73 0     100   100   29     29     500   .23  0  
loops/count_up_down-1.yml .5   18    17    16 110   .20 0     50.0 72.7 .12  .12  17   .078 0  
loops/count_up_down-2.yml .0   .15 .17 11 1.9 .19 0     0   0   .060 .061 9.0 0     0  
loops/eureka_01-1.yml .923 75    69    18 240   .30 0     92.3 100   .21  .20  16   .52  0  
loops/for_bounded_loop1.yml 1.0   43    40    17 170   .29 0     100   100   .20  .24  16   .42  0  
loops/insertion_sort-1.yml 1.0   34    33    17 390   .31 0     100   100   .19  .21  17   .39  0  
loops/insertion_sort-2.yml .9   440    440    18 5800   .39 0     90.0 94.1 10     10     19   1.2   0  
loops/invert_string-1.yml .0   2.1  2.2  10 29   .23 0     0   0   .062 .062 9.0 0     0  
loops/invert_string-3.yml .875 31    29    16 79   .21 0     87.5 94.4 .11  .11  16   .078 0  
loops/linear_sea.ch.yml .667 40    37    18 110   .29 0     66.7 87.5 .20  .20  16   .44  0  
loops/linear_search.yml .929 21    19    18 69   .29 0     92.9 100   .20  .24  17   .44  0  
loops/matrix-1.yml .75  20    18    16 87   .21 0     75.0 94.1 .12  .12  16   .078 0  
loops/matrix-2.yml 1.0   900    900    210 12000   1.2  0     100   100   190     190     1100   1.5   0  
loops/n.c40.yml .5   9.2  8.4  15 38   .20 0     50.0 63.6 .12  .12  16   .094 0  
loops/nec20.yml .833 10    9.4  15 45   .28 0     83.3 95.8 .13  .13  16   .14  0  
loops/nec40.yml .4   9.5  8.7  15 36   .21 0     40.0 46.7 .12  .12  17   .094 0  
loops/string-1.yml .885 92    86    19 250   .28 0     88.5 96.5 .18  .21  17   .38  0  
loops/string-2.yml .962 120    110    20 290   .29 0     96.2 100   .22  .25  16   .44  0  
loops/sum01-1.yml .9   21    20    19 62   .28 .34  90.0 100   3.5   3.6   17   .24  0  
loops/sum01-2.yml .417 13    11    16 100   .20 0     41.7 80.0 .11  .11  16   .078 0  
loops/sum01_bug02.yml .9   540    540    11 6700   .37 0     90.0 100   23     23     16   .36  0  
loops/sum01_bug02_sum01_bug02_base.case.yml 1.0   780    780    10 9100   .39 0     100   100   9.6   9.6   17   .26  0  
loops/sum03-1.yml .0   .15 .16 10 1.6 .20 0     0   0   .060 .061 9.0 0     0  
loops/sum_array-1.yml 1.0   700    710    150 8500   .36 0     100   100   110     110     1000   .78  0  
loops/sum_array-2.yml .929 900    900    150 12000   .81 0     92.9 94.4 110     110     1200   1.2   0  
loops/terminator_01.yml .0   .13 .15 11 1.5 .19 0     0   0   .059 .060 9.0 0     0  
loops/terminator_02-1.yml .0   .14 .16 10 1.6 .20 0     0   0   .067 .068 9.0 0     0  
loops/terminator_02-2.yml .8   190    180    20 610   .29 .061 80.0 94.7 .21  .22  16   .39  0  
loops/terminator_03-1.yml .75  10    9.8  16 110   .20 0     75.0 100   .12  .12  17   .11  0  
loops/terminator_03-2.yml .714 690    680    17 10000   .32 0     71.4 92.3 1.8   1.8   16   .23  0  
loops/trex01-1.yml .0   1.1  1.2  14 18   .20 .061 0   0   .057 .059 9.0 0     0  
loops/trex01-2.yml .846 350    350    56 4000   .29 0     84.6 96.5 26     26     1000   .51  0  
loops/trex02-1.yml .833 900    910    40 13000   .50 0     83.3 92.9 15     15     990   .27  0  
loops/trex02-2.yml .833 900    910    42 12000   .43 0     83.3 100   14     15     990   .29  0  
loops/trex03-1.yml .944 900    910    70 11000   .29 0     94.4 100   38     38     1100   .27  0  
loops/trex03-2.yml .778 900    910    83 12000   .78 0     77.8 93.8 49     49     1100   .49  0  
loops/veris.c_sendmail_tTflag_arr_one_loop.yml .5   33    30    16 87   .20 0     50.0 73.7 .12  .14  16   .078 0  
loops/vogal-1.yml .95  39    36    17 74   .29 0     95.0 96.2 .19  .22  16   .27  0  
loops/vogal-2.yml 1.0   150    140    19 540   .29 0     100   100   .39  .36  17   1.1   0  
loop-acceleration/array_3-1.yml .5   92    92    11 610   .21 0     50.0 72.7 .14  .14  17   .078 0  
loop-acceleration/array_3-2.yml .75  320    320    11 2000   .29 0     75.0 88.9 .57  .57  17   .31  0  
loop-acceleration/array_4.yml .667 160    160    10 1200   .21 0     66.7 90.0 .14  .14  17   .078 0  
loop-acceleration/diamond_1-1.yml .667 29    27    17 81   .20 0     66.7 83.3 .11  .11  17   .078 0  
loop-acceleration/diamond_1-2.yml .0   .15 .17 11 1.7 .20 0     0   0   .076 .077 9.0 0     0  
loop-acceleration/diamond_2-2.yml .542 29    27    17 94   .20 0     54.2 62.1 .12  .12  16   .078 0  
loop-acceleration/multivar_1-1.yml .75  20    18    16 80   .20 0     75.0 90.9 .12  .14  16   .078 0  
loop-acceleration/multivar_1-2.yml .0   .13 .15 10 1.4 .19 0     0   0   .063 .064 9.0 0     0  
loop-acceleration/simple_3-2.yml .75  15    13    15 76   .28 0     75.0 90.0 .17  .16  17   .32  0  
loop-crafted/simple_array_index_value_2.yml .0   3.1  3.2  10 46   .20 .061 0   0   .058 .058 9.0 0     0  
loop-crafted/simple_array_index_value_4.i.v+lhb-reducer.yml .588 78    72    18 430   .29 0     58.8 68.0 .18  .19  16   .37  0  
loop-crafted/simple_array_index_value_4.i.v+nlh-reducer.yml .714 80    74    17 210   .29 0     71.4 77.3 .16  .17  16   .33  0  
loop-crafted/simple_array_index_value_4.yml .75  900    900    11 12000   .59 0     75.0 95.2 .15  .15  16   .23  0  
loop-invgen/MADWiFi-encode_ie_ok.yml .682 37    34    17 170   .27 0     68.2 96.3 .16  .24  17   .25  0  
loop-invgen/NetBSD_loop.yml .125 29    27    17 75   .20 0     12.5 25.0 .12  .12  17   .078 0  
loop-invgen/SpamAssassin-loop.i.v+cfa-reducer.yml .636 900    910    85 13000   .91 .061 63.6 73.3 71     71     1000   .90  0  
loop-invgen/apache-escape-absolute.i.v+cfa-reducer.yml .684 900    900    81 12000   .85 0     68.4 72.7 49     49     1100   1.3   0  
loop-invgen/apache-escape-absolute.yml .833 900    900    85 12000   .93 0     83.3 98.1 51     52     940   .91  0  
loop-invgen/apache-get-tag.i.p+lhb-reducer.yml .513 900    910    68 12000   1.0  0     51.3 59.5 39     39     1000   2.1   0  
loop-invgen/apache-get-tag.i.p+nlh-reducer.yml .618 900    910    81 12000   1.2  0     61.8 74.2 50     51     1100   11     0  
loop-invgen/apache-get-tag.i.p+sep-reducer.yml .430 900    910    100 12000   1.5  0     43.0 48.3 71     71     1100   20     0  
loop-invgen/apache-get-tag.i.v+lhb-reducer.yml .617 900    910    150 13000   .95 0     61.7 69.8 100     100     1100   1.5   0  
loop-invgen/apache-get-tag.i.v+nlh-reducer.yml .625 900    910    140 11000   .88 0     62.5 70.0 110     110     1000   1.5   0  
loop-invgen/apache-get-tag.yml .889 900    910    140 12000   .92 0     88.9 90.7 98     98     1200   .87  0  
loop-invgen/down.yml .833 210    210    11 3200   .33 .59  83.3 94.1 29     29     17   .34  0  
loop-invgen/fragtest_simple.yml .786 170    160    18 460   .30 0     78.6 96.2 .31  .35  16   .77  0  
loop-invgen/half_2.yml .875 42    39    16 100   .29 0     87.5 94.4 .17  .16  16   .34  0  
loop-invgen/heapsort.yml .9   900    910    110 13000   .78 0     90.0 97.6 74     74     1000   .75  0  
loop-invgen/id_build.i.p+nlh-reducer.yml .288 900    900    11 13000   .56 .061 28.8 41.2 55     55     22   2.4   0  
loop-invgen/id_build.i.p+sep-reducer.yml .288 900    900    11 12000   .62 0     28.8 39.8 23     23     22   2.4   0  
loop-invgen/id_build.i.v+lhb-reducer.yml .714 900    900    10 12000   .56 0     71.4 77.8 51     51     16   .39  0  
loop-invgen/id_build.yml .833 900    900    11 13000   .43 0     83.3 90.9 200     200     17   .47  0  
loop-invgen/id_trans.yml 1.0   68    63    19 230   .29 0     100   100   .21  .21  16   .42  0  
loop-invgen/large_const.yml .667 10    9.2  16 57   .28 0     66.7 92.6 .12  .12  16   .11  0  
loop-invgen/nest-if3.yml .857 720    720    100 8600   .32 0     85.7 93.3 270     270     1000   .52  0  
loop-invgen/nested6.yml .857 150    150    36 1000   .29 0     85.7 89.5 12     13     990   .34  0  
loop-invgen/nested9.yml .636 900    900    15 10000   .96 0     63.6 94.4 290     290     16   1.4   0  
loop-invgen/sendmail-close-angle.yml .75  34    32    19 180   .30 0     75.0 93.3 .18  .17  17   .39  0  
loop-invgen/seq-3.yml .812 21    20    15 74   .30 0     81.2 95.7 .18  .20  17   .37  0  
loop-invgen/up.yml .333 28    26    17 78   .20 0     33.3 43.8 .12  .12  17   .078 0  
loop-lit/afnp2014.yml .667 45    42    15 240   .27 0     66.7 91.7 .17  .17  16   .31  0  
loop-lit/bhmr2007.yml .8   900    910    120 13000   .79 0     80.0 93.8 86     86     1000   .81  0  
loop-lit/cggmp2005.yml .0   15    13    22 60   .20 0     0   0   .062 .063 9.0 0     0  
loop-lit/cggmp2005_variant.yml .125 26    24    18 91   .20 0     12.5 26.7 .12  .14  17   .078 0  
loop-lit/cggmp2005b.yml .0   12    10    21 54   .21 0     0   0   .060 .061 9.0 0     0  
loop-lit/css2003.yml .643 18    17    14 81   .21 0     64.3 93.3 .13  .13  16   .094 0  
loop-lit/ddlm2013.yml .9   22    20    15 56   .27 0     90.0 95.5 .15  .14  17   .23  0  
loop-lit/gj2007.c.i.p+lhb-reducer.yml .0   18    16    22 46   .20 0     0   0   .059 .060 9.1 0     0  
loop-lit/gj2007.c.i.p+nlh-reducer.yml .0   19    17    23 58   .21 0     0   0   .062 .062 9.0 0     0  
loop-lit/gj2007.yml .0   16    14    24 57   .20 0     0   0   .060 .063 9.0 0     0  
loop-lit/gj2007b.yml .643 900    910    100 12000   .44 0     64.3 93.3 65     65     1100   .52  0  
loop-lit/gr2006.yml .0   18    16    20 58   .20 0     0   0   .059 .059 9.0 0     0  
loop-lit/gsv2008.c.i.p+cfa-reducer.yml .75  62    57    20 160   .27 0     75.0 85.0 .15  .15  17   .21  0  
loop-lit/gsv2008.c.i.v+cfa-reducer.yml .75  45    40    18 260   .27 .061 75.0 85.0 .14  .14  17   .21  0  
loop-lit/gsv2008.c.i.v+lhb-reducer.yml .7   37    34    17 110   .27 0     70.0 85.7 .15  .14  16   .23  0  
loop-lit/gsv2008.yml .75  22    20    17 89   .20 0     75.0 92.3 .12  .12  16   .094 0  
loop-lit/hhk2008.yml .7   11    11    16 38   .27 .061 70.0 93.8 .14  .14  16   .11  0  
loop-lit/jm2006.c.i.v+cfa-reducer.yml .9   19    17    15 62   .27 0     90.0 92.0 4.6   4.6   17   .16  0  
loop-lit/jm2006.yml .9   17    16    16 60   .27 0     90.0 93.8 3.3   3.4   16   .13  0  
loop-lit/jm2006_variant.yml .75  8.4  7.7  16 32   .28 0     75.0 94.7 .13  .13  16   .13  0  
loop-new/count_by_1.yml .0   17    15    24 60   .20 0     0   0   .061 .061 9.0 0     0  
loop-new/count_by_1_variant.yml .0   13    12    24 50   .20 0     0   0   .058 .058 9.0 0     0  
loop-new/count_by_2.yml .0   16    14    23 47   .20 0     0   0   .059 .060 9.0 0     0  
loop-new/count_by_k.yml .75  15    14    15 67   .27 0     75.0 90.0 .12  .12  16   .11  0  
loop-new/count_by_nondet.yml .75  20    18    18 83   .27 0     75.0 92.9 .15  .15  16   .21  0  
loop-new/gauss_sum.i.p+cfa-reducer.yml .75  16    15    14 76   .28 .061 75.0 85.0 .18  .17  16   .39  0  
loop-new/gauss_sum.i.p+lhb-reducer.yml .611 45    41    17 130   .27 0     61.1 80.0 .14  .14  16   .21  0  
loop-new/gauss_sum.i.v+cfa-reducer.yml .75  15    14    16 87   .28 .074 75.0 85.0 .19  .18  16   .36  0  
loop-new/gauss_sum.yml .75  16    14    17 82   .28 0     75.0 91.7 .20  .19  16   .41  0  
loop-new/half.yml .786 27    25    18 78   .28 0     78.6 92.9 .17  .16  16   .34  0  
loop-new/nested-1.yml .786 97    89    19 300   .28 0     78.6 93.3 .23  .22  16   .31  0  
loop-industry-pattern/mod3.c.v+cfa-reducer.yml .917 18    16    16 99   .27 0     91.7 93.5 .17  .16  16   .29  0  
loop-industry-pattern/mod3.c.v+lhb-reducer.yml .8   450    410    20 2500   .43 0     80.0 86.7 .22  .21  16   .53  0  
loop-industry-pattern/mod3.c.v+sep-reducer.yml .8   140    130    18 440   .29 0     80.0 84.7 .18  .18  16   .41  0  
loop-industry-pattern/mod3.yml .917 50    47    16 140   .27 0     91.7 94.1 .19  .18  17   .37  0  
sv-benchmarks/c/ status score program test-suite TS size inspect test-suite cpu (s) wall (s) mem (MB) energy (J) blkio-w (MB) blkio-r (MB) status program test-suite TS size inspect test-suite cov branch cov (%) line cov (%) cpu (s) wall (s) mem (MB) energy blkio-w (MB) blkio-r (MB)
total 117 73.0 31000 31000 3800 380000 43   1.5  117 7300 8470 2400 2500 25000 75 0  
Run set fairfuzz.test-comp19_prop-coverage-branches.ReachSafety-Loops tbf-testsuite-validator-fairfuzz.test-comp19_prop-coverage-branches.ReachSafety-Loops