Scaling from [Oldfield2014]

This document demonstrates taking some of the scaling data from Oldfield, Moreland, Fabian, and Rogers (2014) and presenting it to show the scaling performance well. These plots demonstrate the metrics outlined in "Formal Metrics for Large-Scale Parallel Performance" by Moreland and Oldfield.

Loading the Data

First the preliminaries. Here are the Python modules we depend on.

In [1]:
import numpy
import pandas
import toyplot.pdf

Read in the raw data file Oldfield2014Raw.csv. This is timing data collected over multiple experiments and averaged together. (We could have done the averaging here, but these are the most accessible data we have.) There are several data that we don't need, but the relevent data is in the following columns:

  • Dataset: A string identifier for the model used by the simulation.
  • cores: The number of compute cores used in the simulation.
  • viz_mean: The total time, in seconds, taken for visualization during the simulation (averaged over all trials).
In [2]:
data = pandas.read_csv("Oldfield2014Raw.csv")
data
Out[2]:
Unnamed: 0 Dataset cores total_mean total_err amrini_mean amrini_err viz_init_mean viz_init_err cth_mean ... viz_mean viz_err viz_sync_data_mean viz_sync_data_err viz_sync_md_mean viz_sync_md_err viz_wait_mean viz_wait_err spy_file_out_mean spy_file_out_err
0 1 33k blocks 128 4246.150670 36.714870 288.832031 0.797634 24.975258 24.808972 3815.476305 ... 116.867076 1.427878 NaN NaN NaN NaN NaN NaN NaN NaN
1 2 33k blocks 256 2501.493490 66.560228 180.370898 2.280082 26.364440 27.740366 2204.032695 ... 90.725456 13.959530 NaN NaN NaN NaN NaN NaN NaN NaN
2 3 33k blocks 512 1407.571875 55.159319 114.635078 0.987830 30.304949 30.891166 1202.640129 ... 59.991719 13.399713 NaN NaN NaN NaN NaN NaN NaN NaN
3 4 33k blocks 1024 867.200195 62.169656 85.012812 4.288807 39.381543 9.162788 692.708008 ... 50.097832 27.537993 NaN NaN NaN NaN NaN NaN NaN NaN
4 5 220k blocks 1024 4174.669596 226.529132 231.158854 10.956432 56.514242 30.825284 3643.094157 ... 243.902344 59.889478 NaN NaN NaN NaN NaN NaN NaN NaN
5 6 220k blocks 2048 2448.479004 135.240054 160.713623 12.372375 107.600016 65.226440 2015.651286 ... 164.514079 21.048313 NaN NaN NaN NaN NaN NaN NaN NaN
6 7 220k blocks 4096 1560.082520 56.162460 113.437061 1.484905 187.804248 51.652706 1137.680859 ... 121.160352 4.799235 NaN NaN NaN NaN NaN NaN NaN NaN
7 8 220k blocks 8192 1159.199524 22.995058 95.841522 0.973975 271.818542 6.967784 703.417816 ... 88.121643 8.844719 NaN NaN NaN NaN NaN NaN NaN NaN
8 9 1.5m blocks 4096 7946.203613 78.314971 368.920085 1.427430 172.924357 47.328550 6236.556437 ... 1167.802734 23.462367 NaN NaN NaN NaN NaN NaN NaN NaN
9 10 1.5m blocks 8192 5389.420573 168.662807 245.578817 15.680281 680.435181 205.710469 3569.504191 ... 893.902384 5.517116 NaN NaN NaN NaN NaN NaN NaN NaN
10 11 1.5m blocks 16384 3641.812134 63.188069 181.351318 1.155300 588.095551 41.348804 2131.576996 ... 740.788269 16.514405 NaN NaN NaN NaN NaN NaN NaN NaN
11 12 1.5m blocks 32768 3496.520996 391.064088 158.363495 4.370822 1249.697876 85.940526 1449.622192 ... 638.837433 14.481826 NaN NaN NaN NaN NaN NaN NaN NaN

12 rows × 21 columns

Our scaling metrics require us to have a measurement of the data size for each experiment. The three data sets for our experiments are identified by the strings '33k blocks', '220k blocks', and '1.5m blocks'. The actual sizes for these data sets in terms of blocks are 33094, 218510, and 1498866, respectively. (All blocks are the same size, so blocks is a valid unit for size of data.) Create a new column for the size of the data in each experiment.

In [3]:
data_set_sizes = { '33k blocks': 33094, '220k blocks': 218510, '1.5m blocks': 1498866 }
blocks_array = numpy.empty(data['Dataset'].size)
for name, blocks in data_set_sizes.iteritems():
    blocks_array[numpy.array(data['Dataset']==name)] = blocks
data['blocks'] = blocks_array

Derived Metrics

The Moreland and Oldfield paper prescribes creating derived quantities for cost, rate and efficiency. We can do this with simple array math.

The rate is the size of the data (in this case, number of blocks) divided by the time it took to compute it.

In [4]:
data['rate'] = data['blocks'] / data['viz_mean']

The cost is the amount of time the run takes times the number of processors used. The cost per unit is the cost divided by the size of the problem.

In [5]:
data['cost'] = data['viz_mean'] * data['cores']
data['cost_per_unit'] = data['cost'] / data['blocks']

The most efficient run has the smallest cost per unit. The efficiency of all other runs is the ratio of the best cost per unit to the actual cost per unit.

In [6]:
best_cost_per_unit = data['cost_per_unit'].min()
data['efficiency'] = best_cost_per_unit / data['cost_per_unit']

We can also use the best cost per unit to determine the ideal rate. It is the number of cores divided by the best cost per unit.

In [7]:
data['ideal_rate'] = data['cores'] / best_cost_per_unit

Traditional Plot

First we create a plot that is commonly seen in performance studies. In this plot we chart number of cores to run time. It is also common to show this on a log-log plot to show multiple scales and/or to make it easier to see hyperbolic curves.

The way we are making our plots is to use a pivot table to convert a table of measurements to a table of series ready to be plotted. (Note that the aggfunc really does not matter here. The data is such that there is only one unique entry (at most) for each pivot table cell, so sum, min, max, and mean will all be the same thing.)

In [8]:
traditional = data.pivot_table(index=['cores'], \
                               columns=['Dataset'], \
                               values='viz_mean', \
                               aggfunc=numpy.mean)
In [9]:
canvas = toyplot.Canvas(400, 320)
axes = canvas.axes(xscale='log2', \
                   yscale='log10', \
                   xlabel='Number of Cores', \
                   ylabel='Time (Seconds)')
axes.x.ticks.locator = toyplot.locator.Explicit(2 ** numpy.arange(7,16))
axes.y.ticks.locator = toyplot.locator.Explicit([100, 1000])
axes.x.ticks.show = True
axes.y.ticks.show = True

x = traditional.index.values
y = numpy.column_stack((traditional['33k blocks'].values, \
                        traditional['220k blocks'].values, \
                        traditional['1.5m blocks'].values))
axes.plot(x, y, marker='o', size=40)
axes.text(1024+100, traditional['33k blocks'][1024], '33k blocks',
          style={'text-anchor':'start'})
axes.text(8192, traditional['220k blocks'][8192], '220k blocks',
          style={'baseline-shift':'-90%'})
axes.text(32768, traditional['1.5m blocks'][32768], '1.5m blocks',
          style={'text-anchor':'end','baseline-shift':'-90%'})
Out[9]:
<toyplot.mark.Text at 0x10de7b5d0>
33k blocks220k blocks1.5m blocks12825651210242048409681921638432768Number of Cores1001000Time (Seconds)

And let's save the plot as OldfieldTimeLog.pdf.

In [10]:
toyplot.pdf.render(canvas, 'OldfieldTimeLog.pdf')
In [11]:
canvas = toyplot.Canvas(400, 320)
axes = canvas.axes(xscale='linear', \
                   yscale='linear', \
                   xlabel='Number of Cores', \
                   ylabel='Time (Seconds)')
#axes.x.ticks.locator = toyplot.locator.Explicit(2 ** numpy.arange(10,14))
#axes.x.ticks.locator = toyplot.locator.Explicit(numpy.arange(0, 8192+1, 8192/8))
axes.x.ticks.locator = toyplot.locator.Explicit([128, 1024, 2048, 4096, 8192])
axes.x.domain.min = 0
axes.x.domain.max = 8192
axes.y.domain.min = 0
axes.x.ticks.show = True
axes.y.ticks.show = True

x = traditional.index.values
y = numpy.column_stack((traditional['33k blocks'].values,
                        traditional['220k blocks'].values))
axes.plot(x, y, marker='o', size=40)
axes.text(1024+150, traditional['33k blocks'][1024], '33k blocks',
          style={'text-anchor':'start'})
axes.text(8192, traditional['220k blocks'][8192], '220k blocks',
          style={'text-anchor':'end', 'baseline-shift':'-90%'})
Out[11]:
<toyplot.mark.Text at 0x10def4250>
33k blocks220k blocks1281024204840968192Number of Cores050100150200250Time (Seconds)

Save the plot as OldfieldTimeLinear.pdf.

In [12]:
toyplot.pdf.render(canvas, 'OldfieldTimeLinear.pdf')

Rate Plot

We've already computed the appropriate rate and ideal rate. Now we just plot them.

In [13]:
rate = data.pivot_table(index=['cores'], \
                        columns=['Dataset'], \
                        values='rate', \
                        aggfunc=numpy.mean)
In [14]:
canvas = toyplot.Canvas(400, 320)
axes = canvas.axes(xlabel='Number of Cores', \
                   ylabel='Rate (Blocks/Second)')
axes.x.ticks.locator = toyplot.locator.Explicit([128, 8192, 16384, 32768])
axes.y.domain.min = 0
axes.y.domain.max = 2600
axes.x.ticks.show = True
axes.y.ticks.show = True

x = rate.index.values
y = numpy.column_stack((rate['33k blocks'].values, \
                        rate['220k blocks'].values, \
                        rate['1.5m blocks'].values))
axes.plot(x, y, marker='o', size=40)
axes.text(1024+500, rate['33k blocks'][1024], '33k blocks',
          style={'text-anchor':'start'})
axes.text(8192, rate['220k blocks'][8192], '220k blocks',
          style={'baseline-shift':'90%'})
axes.text(32768, rate['1.5m blocks'][32768], '1.5m blocks',
          style={'text-anchor':'end','baseline-shift':'90%'})

axes.plot(data['cores'], data['ideal_rate'], \
          style={'stroke':'gray', 'stroke-width':0.5, 'stroke-dasharray':'5,5'})
axes.text(3200, 3000, 'Ideal', fill='gray');
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-abe59cf7baa7> in <module>()
     18 
     19 axes.plot(data['cores'], data['ideal_rate'],           style={'stroke':'gray', 'stroke-width':0.5, 'stroke-dasharray':'5,5'})
---> 20 axes.text(3200, 3000, 'Ideal', fill='gray');

TypeError: text() got an unexpected keyword argument 'fill'
33k blocks220k blocks1.5m blocks12881921638432768Number of Cores0100020003000Rate (Blocks/Second)

Save the plot as OldfieldRate.pdf.

In [ ]:
toyplot.pdf.render(canvas, 'OldfieldRate.pdf')

Efficiency Plot

In [ ]:
efficiency = data.pivot_table(index=['cores'], \
                              columns=['Dataset'], \
                              values='efficiency', \
                              aggfunc=numpy.mean)
In [ ]:
canvas = toyplot.Canvas(400, 320)
axes = canvas.axes(xlabel='Number of Cores', \
                   ylabel='Efficiency')
axes.x.ticks.locator = toyplot.locator.Explicit([128, 8192, 16384, 32768])
axes.x.ticks.show = True
axes.y.ticks.show = True

x = efficiency.index.values
y = numpy.column_stack((efficiency['33k blocks'].values, \
                        efficiency['220k blocks'].values, \
                        efficiency['1.5m blocks'].values))
axes.plot(x, y, marker='o', size=40)
axes.text(512+500, efficiency['33k blocks'][512], '33k blocks',
          style={'text-anchor':'start'})
axes.text(8192, efficiency['220k blocks'][8192], '220k blocks',
          style={'text-anchor':'start','baseline-shift':'90%'})
axes.text(32768, efficiency['1.5m blocks'][32768], '1.5m blocks',
          style={'text-anchor':'end','baseline-shift':'90%'})

Save the plot as OldfieldEfficiency.pdf.

In [ ]:
toyplot.pdf.render(canvas, 'OldfieldEfficiency.pdf')