3.2
· · · · · · · · · ·

The Benchmarks

The current version of PMB-MPI1 contains the benchmarks PingPong PingPing Sendrecv Exchange Bcast Allgather Allgatherv Alltoall Reduce Reduce_scatter

2000-03-09

6 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

· ·

Allreduce Barrier

The exact definitions will be given in section 4. Section 5 describes the benchmark methodology. PMB-MPI1 allows for running all benchmarks in more than one process group. E.g., when running PingPong on N4 processes, on user request (see 5.1.2.3) N/2 disjoint groups of 2 processes each will be formed, all and simultaneously running PingPong. Note that these multiple versions have to be carefully distinguished from their standard equivalents. They will be called
· · · · · · · · · · · ·

Multi-PingPong Multi-PingPing Multi-Sendrecv Multi-Exchange Multi-Bcast Multi-Allgather Multi-Allgatherv Multi-Alltoall Multi-Reduce Multi-Reduce_scatter Multi-Allreduce Multi-Barrier

For a distinction, sometimes we will refer to the standard (non Multi) benchmarks as primary benchmarks. The way of interpreting the timings of the Multi-benchmarks is quite easy, given a definition for the primary cases: per group, this is as in the standard case. Finally, the max timing (min throughput) over all groups is displayed. On request, all per group information can be reported, see 5.1.2.3, 6.3.

3.3
3.3.1
· ·

Version changes
Version 2.1 vs. 2.0
Alltoall added (see 4.4.6) Optional settings mode included (see 5.2.1.)

3.3.2

Version 2.2 vs. 2.1

Default variable initializations (function set_default) were added (March 2000).

3.4

PMB-MPI1 vs. PMB1.x Definitions

Compared to older PMB1.x releases, all primary benchmark names except Sendrecv and Alltoall are the same. Cshift and Xover of PMB1.x have been removed. PMB1.x did not support the Multi versions.

2000-03-09

7 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

Most important is that certain definitions have changed. Please check carefully. Table 2 shows an overview of the changes.

3.4.1

Changed Definitions

The main changes are in PingPing and Exchange. PingPing even uses a different pattern (elementary messages rather than Sendrecv, see 4.3.1). Moreover, the scaling of timings and throughput data has changed.
Timings PMB-MPI1 PMB1.x 0 .5 0 .2 5 Throughputs PMB-MPI1 1 4 PMB1.x 2 4

PingPing Exchange

1 1

Table 1: Scaling factors PMB-MPI1 vs. PMB1.x Thus, the corresponding tables show different values when comparing PMB1.x and PMB-MPI1 on a particular system. The PMB1.x scaling factors for the timings gives confusing answers for the startup components (small message sizes). The scaling of PingPing throughputs by 2 is reasonable (bi-directional throughput), however PMB imposes a different interpretation, leaving the bi-directional throughput measurement to the Sendrecv benchmark. Sendrecv and Alltoall are new benchmarks. Functionally, Sendrecv exactly corresponds to the Cshift benchmark of PMB1.x, however, displays timings with a different scaling. The Xover benchmark of PMB1.x has been removed, as it has shown no significant information on any tested system.

3.4.2

Throughput Calculations

Throughput results are based on real MBytes (1048576 bytes) in PMB-MPI1, in contrast to PMB1.x, which used 1 MByte = 1000000 bytes. In contrast to PMB1.x, PMB-MPI1 does not display throughput values for the global operations Bcast, Allgather and Allgatherv.

3.4.3

Corrected Methodology

PMB1.x was not cleanly defined in the case that a certain benchmark was run in a process group strictly smaller than the group of all started MPI processes. In PMB-MPI1, all non active processes will wait for the active ones in an MPI_Barrier(MPI_COMM_WORLD). In PMB1.x, non active processes immediately went through the outputcollecting phase (MPI_Gather) and then, eventually, switched to the following benchmark(s). This may induce unpredictable obstructions of the active processes. The MPI_Barrier may also, but now the way is well defined and reasonable. See 5 for precise definitions of the methodology.

2000-03-09

8 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

PMB-MPI1 Benchmark name

Contained in releases 1.x â

Compared to PMB1.x, in PMB-MPI1 there is slight change in throughput data due to re-definition of 1MByte = 1048576 bytes other pattern, no scaling expectation: timings doubled, throughputs halved

PingPong

PingPing

â

Sendrecv Exchange Bcast Allgather Allgatherv Alltoall Reduce Reduce_scatter Allreduce Barrier
â â â â no change no change no change no change â â â â 4 fold timings, equal throughputs no output of throughput data no output of throughput data no output of throughput data

PMB1.x benchmarks that are no longer in PMB-MPI1

Cshift Xover

Sendrecv benchmark is a full substitute
Has shown no significant information

Table 2: PMB-MPI1 vs. PMB1.x benchmarks

4

PMB-MPI1 Benchmark Definitions

In this chapter, the single benchmarks are described. Here we focus on the elementary patterns of the benchmarks. The methodology of measuring these patterns (message lengths, sample repetition counts, timer, synchronization, number of processes and communicator management, display of results) are defined in chapters 5 and 6.

4.1

Benchmark Classification

For a clear structuring of the set of benchmarks, PMB now introduces classes of benchmarks: Single Transfer, Parallel Transfer, and Collective. This classification refers to different ways of interpreting results, and to a structuring of the code itself. It does not actually influence the way of using PMB. Also holds this classification hold for the PMB-MPI2 part [3].

2000-03-09

9 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

PMB-MPI1 Single Transfer Parallel Transfer Collective

PingPong PingPing

Sendrecv Exchange

Bcast Allgather Allgatherv

Multi-PingPong Multi-PingPing Multi-Sendrecv Multi-Exchange

Alltoall Reduce Reduce_scatter Allreduce Barrier
Multi-versions of these

4.1.1

Single Transfer Benchmarks

The benchmarks in this class are to focus on a single message transferred between two processes. As to PingPong, this is the usual way of looking at. In PMB interpretation, PingPing measures the same as PingPong, under the particular circumstance that a message is obstructed by an oncoming one (sent simultaneously by the same process that receives the own one). Single transfer benchmarks, roughly speaking, are local mode. The particular pattern is purely local to the participating processes, there is no concurrency with other message passing activity. Best case message passing results are to be expected. Important for this is that single transfer benchmarks only run with 2 active processes (see 3.4.3, 5.2.2 for the definition of active). For PingPing, and this is in contrast to PMB1.x and other code systems containing this benchmark, pure timings will be reported, and the throughput is related to a single message. Expected numbers, very likely, are between half and full PingPong throughput. W ith this, PingPing determines the throughput of messages under non optimal conditions (namely, oncoming traffic). See 4.2.1 and 4.2.2 for exact definitions.

4.1.2

Parallel Transfer Benchmarks

Benchmarks focusing on global mode, say, patterns. The activity at a certain process is in concurrency with other processes, the benchmark measures message passing efficiency under global load. For the interpretation of Sendrecv and Exchange, more than 1 message (per sample) counts. As to the throughput numbers, the total turnover (the number of sent plus the number of received bytes ) at a certain process is taken into account. E.g., for the case of 2 processes, Sendrecv becomes the bi-directional test: perfectly bi-directional systems are rewarded by a double PingPong throughput here. Thus, the throughputs are scaled by certain factors. See 4.3.1 and 4.3.2 for exact definitions. As to the timings, raw results without scaling will be reported. The Multi mode secondarily introduces into this class
· · ·

Multi-PingPong Multi-PingPing Multi-Sendrecv

2000-03-09

10 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

·

Multi-Exchange

4.1.3

Collective Benchmarks

This class contains all benchmarks that are collective in proper MPI convention. Not only is the message passing power of the system relevant here, but also the quality of the implementation. For simplicity, we also include the Multi versions of these benchmarks into this class. Raw timings and no throughput are reported. Note that certain collective benchmarks (namely the reductions) play a particular role as they are not pure message passing tests, but also depend on an efficient implementation of certain numerical operations.

4.2

Definition of Single Transfer Benchmarks

This section describes the single transfer benchmarks in detail. Each benchmark is run with varying message lengths X bytes, and timings are averaged over multiple samples. See 5 for the description of the methodology. Here we describe the view of one single sample, with a fixed message length X bytes. Basic MPI datatype for all messages is MPI_BYTE. Throughput values are defined in MBytes / sec = 220 bytes / sec scale (i.e. throughput = X / 220 * 106 / time = X / 1.048576 / time, when time is in µsec).

2000-03-09

11 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

4.2.1

PingPong

PingPong is the classical pattern used for measuring startup and throughput of a single message sent between two processes.

Measured pattern based on

As symbolized between in Figure 1; two active processes only (Q=2, see 5.2.2)

MPI_Send, MPI_Recv MPI_BYTE
time = t/2 (in µsec) as indicated in Figure 1 X/1.048576/time

MPI_Datatype
reported timings reported throughput

PROCESS 1 MPI_Send time=t/2 t X bytes

PROCESS 2

MPI_Recv MPI_Send MPI_Recv X bytes

Figure 1: PingPong pattern

2000-03-09

12 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

4.2.2

PingPing

As PingPong, PingPing measures startup and throughput of single messages, with the crucial difference that messages are obstructed by oncoming messages. For this, two processes communicate (MPI_Isend/MPI_Recv/MPI_Wait) with each other, with the MPI_Isend's issued simultaneously.

Measured pattern

As symbolized between Figure 2;

in

two active processes only (Q=2, 5.2.2) based on

MPI_Isend/MPI_Wait, MPI_Recv MPI_BYTE time = t (in µsec) as indicated in
Figure 2

MPI_Datatype
reported timings

reported throughput

X/1.048576/time

PROCESS 1 MPI_Isend( t MPI_Recv MPI_Wait(R) )

PROCESS 2 MPI_Isend( X bytes MPI_Recv MPI_Wait(R) )

request=R

request=R

X bytes

Figure 2: PingPing pattern

2000-03-09

13 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

4.3

Definition of Parallel Transfer Benchmarks

This section describes the parallel transfer benchmarks in detail. Each benchmark is run with varying message lengths X bytes, and timings are averaged over multiple samples. See 5 for the description of the methodology. Here we describe the view of one single sample, with a fixed message length X bytes. Basic MPI datatype for all messages is MPI_BYTE. The throughput calculations of the benchmarks described here take into account the (per sample) multiplicity nmsg of messages outgoing from or incoming at a particular process. In the Sendrecv benchmark, a particular process sends and receives X bytes, the turnover is 2X bytes, nmsg=2. In the Exchange case, we have 4X bytes turnover, nmsg=4. Throughput values are defined in MBytes/sec = 2 (i.e.
throughput = nmsg*X/2 when time is in µsec).
20 20

bytes / sec scale

* 106/time = nmsg*X / 1.048576 / time,

2000-03-09

14 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

4.3.1

Sendrecv

Based on MPI_Sendrecv, the processes form a periodic communication chain. Each process sends to the right and receives from the left neighbor in the chain. The turnover count is 2 messages per sample (1 in, 1 out) for each process. Sendrecv is equivalent with the Cshift benchmark and, in case of 2 processes, the PingPing benchmark of PMB1.x. For 2 processes, it will report the bi-directional bandwidth of the system, as obtained by the (optimized) MPI_Sendrecv function.

Measured pattern based on

As symbolized between

in Figure 3

MPI_Sendrecv MPI_BYTE
time = t (in µsec) as indicated in Figure 3 2X/1.048576/time

MPI_Datatype
reported timings reported throughput

Periodic chain

...

PR. I-1

PR. I

PR. I+1 ...

t

MPI_ X bytes MPI_ X bytes MPI_ Sendrecv Sendrecv Sendrecv

Figure 3: Sendrecv pattern

2000-03-09

15 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

4.3.2

Exchange

Exchange is a communications pattern that often occurs in grid splitting algorithms (boundary exchanges). The group of processes is seen as a periodic chain, and each process exchanges data with both left and right neighbor in the chain. The turnover count is 4 messages per sample (2 in, 2 out) for each process.
Measured pattern based on As symbolized between in Figure 4

MPI_Isend/MPI_Waitall, MPI_Recv MPI_BYTE time = t (in µsec) as indicated in Figure 4 4X/1.048576/time

MPI_Datatype
reported timings reported throughput

Periodic chain

...

PR. I-1 MP MP MP MP MP I_Isend I_Isend I_Recv I_Recv I_Waitall

PR. I MP MP MP MP MP I_Isend I_Isend I_Recv I_Recv I_Waitall

PR. I+1 MP MP MP MP MP

...

t

I_Isend I_Isend I_Recv I_Recv I_Waitall

Each
Figure 4: Exchange pattern

carries X bytes

4.4

Definition of Collective Benchmarks

This section describes the Collective benchmarks in detail. Each benchmark is run with varying message lengths X bytes, and timings are averaged over multiple samples. See 5 for the description of the methodology. Here we describe the view of one single sample, with a fixed message length X bytes. Basic MPI datatype for all messages is MPI_BYTE for the pure data movement functions, and MPI_FLOAT for the reductions. For all Collective benchmarks, only bare timings and no throughput data is displayed.

2000-03-09

16 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

4.4.1

Reduce

Benchmark of the MPI_Reduce function. Reduces a vector of length L = X/sizeof(float) float items. The MPI datatype is MPI_FLOAT, the MPI operation is MPI_SUM. The root of the operation is changed cyclically, see 5.2.7 See also the remark in the end of 4.1.3.
measured pattern

MPI_Reduce MPI_FLOAT MPI_SUM
changing bare time none

MPI_Datatype MPI_Op
root reported timings reported throughput

4.4.2

Reduce_scatter

Benchmark of the MPI_Reduce_scatter function. Reduces a vector of length L = X/sizeof(float)float items. The MPI datatype is MPI_FLOAT, the MPI operation is MPI_SUM. In the scatter phase, the L items are split as evenly as possible. Exactly, when
np = #processes, L = r*np+s (s = L mod np),

then process with rank i gets r+1 items when i measured pattern

MPI_Reduce_scatter MPI_FLOAT MPI_SUM
bare time none

MPI_Datatype MPI_Op
reported timings reported throughput

4.4.3

Allreduce

Benchmark of the MPI_Allreduce function. Reduces a vector of length L = X/sizeof(float) float items. The MPI datatype is MPI_FLOAT, the MPI operation is MPI_SUM. See also the remark in the end of 4.1.3.
measured pattern

MPI_Allreduce MPI_FLOAT MPI_SUM
bare time none

MPI_Datatype MPI_Op
reported timings reported throughput

2000-03-09

17 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

4.4.4

Allgather

Benchmark of the MPI_Allgather function. Every process inputs X bytes and receives the gathered X*(#processes) bytes.
Measured pattern

MPI_Allgather MPI_BYTE
bare time none

MPI_Datatype
reported timings reported throughput

4.4.5

Allgatherv

Functionally the same as Allgather, however with the MPI_Allgatherv function. Shows whether MPI produces overhead due to the more complicated situation as compared to MPI_Allgather.
Measured pattern

MPI_Allgatherv MPI_BYTE
bare time none

MPI_Datatype
reported timings reported throughput

4.4.6

Alltoall

Benchmark of the MPI_Alltoall function. Every process inputs X*(#processes) bytes (X for each process) and receives X*(#processes) bytes (X from each process).
Measured pattern

MPI_Alltoall MPI_BYTE
bare time none

MPI_Datatype
reported timings reported throughput

4.4.7

Bcast

Benchmark of MPI_Bcast. A root process broadcasts X bytes to all. The root of the operation is changed cyclically, see 5.2.7.
measured pattern MPI_Datatype root reported timings reported throughput

MPI_Bcast MPI_BYTE
changing bare time none

2000-03-09

18 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

4.4.8

Barrier
MPI_Barrier
bare time none

measured pattern reported timings reported throughput

5

Benchmark Methodology

Recall that in chapter 4 only the underlying patterns of each benchmark have been defined. In this section, the measuring method for those patterns is explained. Some control mechanisms are hard coded (like the selection of process numbers to run the benchmarks on), some are set by preprocessor parameters in a central include file. Important is that (in contrast to the previous release 2.0) there is a standard and an optional mode to control PMB. In standard mode, all configurable sizes are predefined and should not be changed. This assures comparability for a result tables in standard mode. In optional mode, the user can set those parameters at own choice. For instance, this mode can be used to extend the results tables as to larger message size. The following graph shows an overview of the flow of control inside PMB. All emphasized items will be explained in more detail. For ( all_selected_benchmarks ) For ( all_selected_process_numbers ) Select MPI communicator MY_COMM to run the benchmark, (see 5.2.2) For ( all_selected_message_lengths X ) (see 5.2.3) Initialize communication buffers (see 5.2.4)
X == first_selected_message_length

Yes No Warm_up (see 5.2.5)
MY_COMM != MPI_COMM_NULL

Yes Synchronize processes of MY_COMM (see 5.2.6) Execute benchmark (message size = X ) (see 5.2.7) MPI_Barrier (MPI_COMM_WORLD) Output results (see 6) Figure 5: Control flow in PMB

No

The control parameters obviously necessary are either command line arguments (see 5.1) or parameter selections inside the PMB include file settings.h (see 5.2).

2000-03-09

19 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

5.1

Running PMB, Command Line Control

After installation, see 2.2, an executable PMB-MPI1 should exist. Given P, the (normally user selected) number of MPI processes to run PMBMPI1, a startup procedure has to load parallel PMB-MPI1. Lets assume, for sake of simplicity, that this done by mpirun ­np P PMB-MPI1 [arguments]
P=1 is allowed, will be ignored only by Single Transfer benchmarks. Control arguments (in addition to P) can be passed to PMB-MPI1 via (argc,argv)

which will be read by PMB-MPI1 process 0 (in MPI_COMM_WORLD ranking) and then distributed to all processes.

5.1.1

Default Case

Just invoke mpirun ­np P PMB-MPI1 All primary (non Multi) benchmarks will run on
Q=2, 4, 8, ..., largest 2x
(E.g P=11, then 2,4,8,11 processes will be selected). The Q<=P processes running the benchmark are called active processes. A communicator is formed out of a group of Q processes, see 5.2.2., which is used as communicator argument to the MPI functions crucial for the benchmark.

5.1.2

Command Line Control

The general syntax is mpirun ­np P PMB-MPI1 [Benchmark1 [Benchmark2 [ ... ] ] ] [-npmin P_min] [-multi Outflag] [-input ] (where the 4 major [ ] may appear in any order). Examples: mpirun ­np 8 PMB-MPI1 mpirun ­np 10 PMB-MPI1 PingPing Reduce mpirun ­np 11 PMB-MPI1 ­npmin 5 mpirun ­np 4 PMB-MPI1 ­npmin 4 ­input PMB_SELECT_MPI1

mpirun ­np 14 PMB-MPI1 ­multi 0 PingPong Barrier ­npmin 7

5.1.2.1 Benchmark Selection Arguments
A set of blank-separated strings, each being the name of one primary (non Multi) PMB benchmark (in exact spelling, case insensitive). Default (no benchmark selection): select all primary benchmark names. Given a name selection, either

2000-03-09

20 of 34


Pallas MPI Benchmarks - PMB, Part MPI-1

-multi flag is missing, in which case all selected primary benchmarks are run, or -multi flag is selected, and then the Multi- versions of all selected benchmarks are executed.

5.1.2.2 ­npmin Selection
The argument after ­npmin has to be an integer P_min, specifying the minimum number of processes to run all selected benchmarks.
· · ·

P_min may be 1 P_min > P is handled as P_mi