MPE - MultiProcessing Environment

The Multi-Processing Environment (MPE) attempts to provide programmers with a complete suite of performance analysis tools for their MPI programs based on post processing approach. These tools include a set of profiling libraries, a set of utility programs, and a set of graphical visualization tools.

The first set of tools to be used with user MPI programs is profiling libraries which provide a collection of routines that create log files. These log files can be created manually by inserting MPE calls in the MPI program, or automatically by linking with the appropriate MPE libraries, or by combining the above two methods. Currently, the MPE offers the following 3 profiling libraries.

Tracing Library

- This traces all MPI calls. Each MPI call is preceded by a line that contains the rank in MPI_COMM_WORLD of the calling process, and followed by another line indicating that the call has completed. Most send and receive routines also indicate the values of count, tag, and partner (destination for sends, source for receives). Output is to standard output.

Animation Library

- This is a simple form of real-time program animation and requires X window routines.

Logging Library

- This is the most useful and widely used profiling libraries in MPE. They form the basis to generate log files from user MPI programs. There are currently 3 different log file formats allowed in MPE. The default log file format is CLOG. It is basically a collection of events with single timestamps. And there is ALOG which is provided for backward compatibility reason and it is not being developed. And the most powerful one is SLOG, stands for Scalable LOGfile format, which can be converted from CLOG after CLOG file has been generated (preferred approach), or can be generated directly when MPI program is executed (through setting the environmental variable MPE_LOG_FORMAT to SLOG).

The set of utility programs in MPE includes log format converter (e.g. clog2slog), logfile print (e.g. slog_print) and logfile viewer wrappers, logviewer, which selects the correct graphical tool to display the logfile based on the logfile's file extension.

Currently, MPE's graphical tools includes 3 display programs, upshot for ALOG, jumpshot-2 for CLOG and jumpshot-3 for SLOG. The logviewer script eliminates the need for user to remember the relationship of logfile formats to display programs.

MPE can be configured and installed as an extension to most MPI standard compliant MPI implementations, e.g. MPICH, LAM, SGI's MPI, HP-UX's MPI and IBM's MPI. It has been integrated seamlessly into MPICH distribution, so MPE will be installed automatically during MPICH's installation process.

For details of configuring and building of MPE, please refer to the README file in the MPE source directory.

As previously noted, the mpe library is composed of 3 different profiling libraries. Each MPI implementation requires a slightly different way in which to link with these libraries. During configure, the link path and appropriate libraries are determined and assigned to variables. These variables are first substituted in the Makefile in the directory mpe/contrib/test . The Makefile is then installed into directory share/examples during the final installation process. The following is a list of these variables.

LOG_LIBS = link path needed to link with the logging library

TRACE_LIBS = link path needed to link with the tracing library

ANIM_LIBS = link path needed to link with the animation library

The variable FLIB_PATH is the link path needed to link fortran MPI programs with the logging library.

During make, small test programs cpi.c and fpi.f will be linked with each of the above libraries. In the output from Make, a message will be written regarding the success of each attempted link test. The success of these linkage tests will also be included in the Make output. If the linkage tests are successful, then these library link paths can be used for your programs as well.

In addition to using the predefined MPE logging libraries to log all MPI calls, MPE logging calls can be inserted into user's MPI program to define and log states. These states are called User-Defined states. States may be nested, allowing one to define a state describing a user routine that contains several MPI calls, and display both the user-defined state and the MPI operations contained within it.

The routine MPE_Log_get_event_number() has to be used to get unique event numbers (this is important if you are writing a library that uses the MPE logging routines) from the MPE system. The routines MPE_Describe_state() and MPE_Log_event() are then used to describe user-defined states.

int eventID_begin, eventID_end;
.
.
.
eventID_begin = MPE_Log_get_event_number();
eventID_end   = MPE_Log_get_event_number();
.
.
.
MPE_Describe_state( eventID_begin, eventID_end, "Amult", "bluegreen" );
.
.
.
MyAmult( Matrix m, Vector v )
{
/* Log the start event along with the size of the matrix */
MPE_Log_event( eventID_begin, m->n, (char *)0 );
.
.
.
Amult code, including MPI calls ...
MPE_Log_event( eventID_end, 0, (char *)0 );
}

The log file generated by this code will have the MPI routines within the routine MyAmult() indicated by a containing bluegreen rectangle.

If the MPE logging library, liblmpe.a, are NOT linked with the user program, MPE_Init_log() and MPE_Finish_log() need to be used before and after all the MPE calls. Sample programs cpilog.c and fpi.f are available in MPE source directory contrib/test or the installed directory share/examples to illustrate the use of these MPE routines.

There are 2 environmental variables, TMPDIR and MPE_LOG_FORMAT , that user

MPE_LOG_FORMAT

- determines the format of the logfile generated from the execution of application linked with MPE logging libraries. The allowed value for MPE_LOG_FORMAT are CLOG, SLOG and ALOG. When MPE_LOG_FORMAT is NOT set, CLOG is assumed.

TMPDIR

- specifies a directory to be used as temporary storage for each process. By default, when TMPDIR is NOT set, /tmp will be used. When user needs to generate a very large logfile for long-running MPI job, user needs to make sure that TMPDIR is big enough to hold the temporary logfile which will be deleted if the merged logfile can be created successfully. In order to minimize the overhead of the logging to the MPI program, it is highly recommended user to use a *local* file system for TMPDIR .

Note : The final merged logfile will be written back to the file system where process 0 is.

In bin/, user can find several useful utility programs when manipulating logfiles. These includes log format converters, e.g. clog2slog, log format print programs, e.g. slog_print, and a script to launch display program, logviewer.

clog2slog

- a CLOG to SLOG logfile converter. Since the automatic generation of SLOG file through setting of environmental variable MPE_LOG_FORMAT to SLOG may NOT work for some non well-behaved MPI programs, using the logfile format converter can generate extra diagnostic information about the condition of the logfile. Also the converter allows one to adjust certain parameters of the logfile, like frame size which is the segment of the logfile to be displayed by Jumpshot-3's time line window. For non well behaved MPI program, one may need to increase the frame size from the default 64KB to a bigger value. For more information about the converter, do "clog2slog -h"

clog2alog

- a CLOG to ALOG logfile converter. It is not being developed. It is provided here for backward compatibility purpose.

slog_print

- a stdout print program for SLOG file. It serves to check the content of the logfile. If the SLOG file is too big, it may NOT be useful to use slog_print. Also, when slog is NOT complete, slog_print won't work. So it serves as a simple test to check if the SLOG file is generated completely.

clog_print

- stdout print program for CLOG file.

logviewer

- the script which involves appropriate viewer based on the file extension of logfile. For instance, if the logfile is foo.slog, logviewer will invoke jumpshot-3 to display the logfile. Jumpshot-3 resides in share/. For more information of logviewer, do "logviewer -help" to list all available options.

To view a list of configure options, type ./configure--help in the top-level mpe directory.

In the Installation Manual, there is a section on MPE which gives instructions and examples for each type of installation.

The README in the top-level mpe directory.

In the Users Guide, there is additional information describing the MPE routines and the profiling libraries.

The man pages included in the distribution has information regarding the MPE routines. The command mpich/bin/mpiman is a script that runs xman on these man pages.

/home/MPI/mansrc/mpepage