This folder contains raw performance data for the case study of incremental lambda calculi, sections 4.4, 4.5 and 4.6. Data gathered from previous experiments are included as is, in the spirit of lab notes.
The goal of this artifact is simply to present the raw data from which the graphs shown in the paper were computed.
Moreover, if you rerun the benchmarks, you will get output in the same format, which is documented here.
Graph included in the paper
Excel file to generate graphs - the relevant sheet is the last one, Graphs-new
. It contains a simple graph computed out of the raw datasets, described below.
HistogramGenerated.xlsx, after conversion into OpenOffice/LibreOffice formats; it can be opened inside the VM image with LibreOffice Calc (the .xlsx
file can also be opened, but is understood less well). The conversion result is not perfect, but you only miss a recomputation in Excel of the confidence intervals, which was not used in the graphs — this shows up through #NAME messages.
This file (in Markdown format).
Performance of incremental computation in the format of semicolon-separated-values. Rows relevant to the case study has “random changes” as the value of the column “param-change”. They begin on line 71 and end on line 83.
Performance of recomputation in the format of semicolon- separated-values. Rows relevant to the case study has “random change” as the value of the column “param-change”. They begin on line 62 and end on line 74.
The benchmarking infrastructure we use is ScalaMeter, and it determines the output format — basically CSV files using semicolon as separators.
Each line corresponds to all the executions of a testcase that are combined and averaged to form a datapoint.
The meaning of each line is described by the heading:
date;param-n;param-change;value;success;cilo;cihi;units;complete
The different fields mean:
An example record is:
2013-11-09T05:42:37Z;4096;random changes;0.15081383333333334;true;0.139;0.162;ms;"0.144769 0.145168 0.154076 0.201024 0.138941 0.139314 0.139763 0.193359 0.124291 0.124329 0.125239 0.189909 0.131056 0.131273 0.131358 0.196586 0.138116 0.138442 0.142803 0.190991 0.135526 0.13669 0.136763 0.193228 0.128335 0.129049 0.129425 0.192945 0.140421 0.140662 0.141121 0.192908 0.133958 0.134542 0.142451 0.200467"