This folder contains raw performance data for the case study of incremental lambda calculi, sections 4.4, 4.5 and 4.6. Data gathered from previous experiments are included as is, in the spirit of lab notes.

The goal of this artifact is simply to present the raw data from which the graphs shown in the paper were computed.

Moreover, if you rerun the benchmarks, you will get output in the same format, which is documented here.

List of Files

HistogramGenerated-new.pdf

Graph included in the paper

HistogramGenerated.xlsx

Excel file to generate graphs - the relevant sheet is the last one, Graphs-new. It contains a simple graph computed out of the raw datasets, described below.

HistogramGenerated.ods

HistogramGenerated.xlsx, after conversion into OpenOffice/LibreOffice formats; it can be opened inside the VM image with LibreOffice Calc (the .xlsx file can also be opened, but is understood less well). The conversion result is not perfect, but you only miss a recomputation in Excel of the confidence intervals, which was not used in the graphs — this shows up through #NAME messages.

README.md

This file (in Markdown format).

testOutput-upd/ilc.examples.HistogramGenerated (derivative, surgical change).Test-0.dsv

Performance of incremental computation in the format of semicolon-separated-values. Rows relevant to the case study has “random changes” as the value of the column “param-change”. They begin on line 71 and end on line 83.

testOutput-upd/ilc.examples.HistogramGenerated (recomputation).Test-1.dsv

Performance of recomputation in the format of semicolon- separated-values. Rows relevant to the case study has “random change” as the value of the column “param-change”. They begin on line 62 and end on line 74.

Format of the .dsv files

The benchmarking infrastructure we use is ScalaMeter, and it determines the output format — basically CSV files using semicolon as separators.

Each line corresponds to all the executions of a testcase that are combined and averaged to form a datapoint.

The meaning of each line is described by the heading:

date;param-n;param-change;value;success;cilo;cihi;units;complete

The different fields mean:

An example record is:

2013-11-09T05:42:37Z;4096;random changes;0.15081383333333334;true;0.139;0.162;ms;"0.144769 0.145168 0.154076 0.201024 0.138941 0.139314 0.139763 0.193359 0.124291 0.124329 0.125239 0.189909 0.131056 0.131273 0.131358 0.196586 0.138116 0.138442 0.142803 0.190991 0.135526 0.13669 0.136763 0.193228 0.128335 0.129049 0.129425 0.192945 0.140421 0.140662 0.141121 0.192908 0.133958 0.134542 0.142451 0.200467"