This document demonstrates taking some of the scaling data from Oldfield, Moreland, Fabian, and Rogers (2014) and presenting it to show the scaling performance well. These plots demonstrate the metrics outlined in "Formal Metrics for Large-Scale Parallel Performance" by Moreland and Oldfield.
First the preliminaries. Here are the Python modules we depend on.
import numpy
import pandas
import toyplot.pdf
Read in the raw data file Oldfield2014Raw.csv. This is timing data collected over multiple experiments and averaged together. (We could have done the averaging here, but these are the most accessible data we have.) There are several data that we don't need, but the relevent data is in the following columns:
data = pandas.read_csv("Oldfield2014Raw.csv")
data
Our scaling metrics require us to have a measurement of the data size for each experiment. The three data sets for our experiments are identified by the strings '33k blocks', '220k blocks', and '1.5m blocks'. The actual sizes for these data sets in terms of blocks are 33094, 218510, and 1498866, respectively. (All blocks are the same size, so blocks is a valid unit for size of data.) Create a new column for the size of the data in each experiment.
data_set_sizes = { '33k blocks': 33094, '220k blocks': 218510, '1.5m blocks': 1498866 }
blocks_array = numpy.empty(data['Dataset'].size)
for name, blocks in data_set_sizes.iteritems():
blocks_array[numpy.array(data['Dataset']==name)] = blocks
data['blocks'] = blocks_array
The Moreland and Oldfield paper prescribes creating derived quantities for cost, rate and efficiency. We can do this with simple array math.
The rate is the size of the data (in this case, number of blocks) divided by the time it took to compute it.
data['rate'] = data['blocks'] / data['viz_mean']
The cost is the amount of time the run takes times the number of processors used. The cost per unit is the cost divided by the size of the problem.
data['cost'] = data['viz_mean'] * data['cores']
data['cost_per_unit'] = data['cost'] / data['blocks']
The most efficient run has the smallest cost per unit. The efficiency of all other runs is the ratio of the best cost per unit to the actual cost per unit.
best_cost_per_unit = data['cost_per_unit'].min()
data['efficiency'] = best_cost_per_unit / data['cost_per_unit']
We can also use the best cost per unit to determine the ideal rate. It is the number of cores divided by the best cost per unit.
data['ideal_rate'] = data['cores'] / best_cost_per_unit
First we create a plot that is commonly seen in performance studies. In this plot we chart number of cores to run time. It is also common to show this on a log-log plot to show multiple scales and/or to make it easier to see hyperbolic curves.
The way we are making our plots is to use a pivot table to convert a table of measurements to a table of series ready to be plotted. (Note that the aggfunc really does not matter here. The data is such that there is only one unique entry (at most) for each pivot table cell, so sum, min, max, and mean will all be the same thing.)
traditional = data.pivot_table(index=['cores'], \
columns=['Dataset'], \
values='viz_mean', \
aggfunc=numpy.mean)
canvas = toyplot.Canvas(400, 320)
axes = canvas.axes(xscale='log2', \
yscale='log10', \
xlabel='Number of Cores', \
ylabel='Time (Seconds)')
axes.x.ticks.locator = toyplot.locator.Explicit(2 ** numpy.arange(7,16))
axes.y.ticks.locator = toyplot.locator.Explicit([100, 1000])
axes.x.ticks.show = True
axes.y.ticks.show = True
x = traditional.index.values
y = numpy.column_stack((traditional['33k blocks'].values, \
traditional['220k blocks'].values, \
traditional['1.5m blocks'].values))
axes.plot(x, y, marker='o', size=40)
axes.text(1024+100, traditional['33k blocks'][1024], '33k blocks',
style={'text-anchor':'start'})
axes.text(8192, traditional['220k blocks'][8192], '220k blocks',
style={'baseline-shift':'-90%'})
axes.text(32768, traditional['1.5m blocks'][32768], '1.5m blocks',
style={'text-anchor':'end','baseline-shift':'-90%'})
And let's save the plot as OldfieldTimeLog.pdf.
toyplot.pdf.render(canvas, 'OldfieldTimeLog.pdf')
canvas = toyplot.Canvas(400, 320)
axes = canvas.axes(xscale='linear', \
yscale='linear', \
xlabel='Number of Cores', \
ylabel='Time (Seconds)')
#axes.x.ticks.locator = toyplot.locator.Explicit(2 ** numpy.arange(10,14))
#axes.x.ticks.locator = toyplot.locator.Explicit(numpy.arange(0, 8192+1, 8192/8))
axes.x.ticks.locator = toyplot.locator.Explicit([128, 1024, 2048, 4096, 8192])
axes.x.domain.min = 0
axes.x.domain.max = 8192
axes.y.domain.min = 0
axes.x.ticks.show = True
axes.y.ticks.show = True
x = traditional.index.values
y = numpy.column_stack((traditional['33k blocks'].values,
traditional['220k blocks'].values))
axes.plot(x, y, marker='o', size=40)
axes.text(1024+150, traditional['33k blocks'][1024], '33k blocks',
style={'text-anchor':'start'})
axes.text(8192, traditional['220k blocks'][8192], '220k blocks',
style={'text-anchor':'end', 'baseline-shift':'-90%'})
Save the plot as OldfieldTimeLinear.pdf.
toyplot.pdf.render(canvas, 'OldfieldTimeLinear.pdf')
We've already computed the appropriate rate and ideal rate. Now we just plot them.
rate = data.pivot_table(index=['cores'], \
columns=['Dataset'], \
values='rate', \
aggfunc=numpy.mean)
canvas = toyplot.Canvas(400, 320)
axes = canvas.axes(xlabel='Number of Cores', \
ylabel='Rate (Blocks/Second)')
axes.x.ticks.locator = toyplot.locator.Explicit([128, 8192, 16384, 32768])
axes.y.domain.min = 0
axes.y.domain.max = 2600
axes.x.ticks.show = True
axes.y.ticks.show = True
x = rate.index.values
y = numpy.column_stack((rate['33k blocks'].values, \
rate['220k blocks'].values, \
rate['1.5m blocks'].values))
axes.plot(x, y, marker='o', size=40)
axes.text(1024+500, rate['33k blocks'][1024], '33k blocks',
style={'text-anchor':'start'})
axes.text(8192, rate['220k blocks'][8192], '220k blocks',
style={'baseline-shift':'90%'})
axes.text(32768, rate['1.5m blocks'][32768], '1.5m blocks',
style={'text-anchor':'end','baseline-shift':'90%'})
axes.plot(data['cores'], data['ideal_rate'], \
style={'stroke':'gray', 'stroke-width':0.5, 'stroke-dasharray':'5,5'})
axes.text(3200, 3000, 'Ideal', fill='gray');
Save the plot as OldfieldRate.pdf.
toyplot.pdf.render(canvas, 'OldfieldRate.pdf')
efficiency = data.pivot_table(index=['cores'], \
columns=['Dataset'], \
values='efficiency', \
aggfunc=numpy.mean)
canvas = toyplot.Canvas(400, 320)
axes = canvas.axes(xlabel='Number of Cores', \
ylabel='Efficiency')
axes.x.ticks.locator = toyplot.locator.Explicit([128, 8192, 16384, 32768])
axes.x.ticks.show = True
axes.y.ticks.show = True
x = efficiency.index.values
y = numpy.column_stack((efficiency['33k blocks'].values, \
efficiency['220k blocks'].values, \
efficiency['1.5m blocks'].values))
axes.plot(x, y, marker='o', size=40)
axes.text(512+500, efficiency['33k blocks'][512], '33k blocks',
style={'text-anchor':'start'})
axes.text(8192, efficiency['220k blocks'][8192], '220k blocks',
style={'text-anchor':'start','baseline-shift':'90%'})
axes.text(32768, efficiency['1.5m blocks'][32768], '1.5m blocks',
style={'text-anchor':'end','baseline-shift':'90%'})
Save the plot as OldfieldEfficiency.pdf.
toyplot.pdf.render(canvas, 'OldfieldEfficiency.pdf')