MPICH-MX
From HP-SEE Wiki
Section contributed by IFIN-HH
MPICH was ported by Myricom [myrc] to the Myrinet cards, on top of the Myrinet Express (MX) API [mymx], which includes the Myrinet proprietary driver ch_mx.
Myricom’s high speed interconnect and protocols, which provide a throughput near the nominal signaling speed of the physical layer, gradually evolved from 2Gbps to 10Gbps. Until the very recent implementation of OpenMPI, the only support for MPI on Myrinet cards was ensured by MPICH-MX, which is based on the version 1.2.7 of the ANL’s MPICH.
Myrinet technology is represented in the HP-SEE infrastructure by IFIN_Bio, which currently supports MPICH-MX only.
Installation
The installation of MPICH-MX is performed under root priviledges. Some software prerequisites are required in order to install the library MPICH-MX: ssh/rsh, MX API, C and Fortran compilers.
MPICH-MX provides the installer script mpich.make.linux, that includes all the information necessary for the configure/make/install steps, and which should be customized to the hosting cluster. Usually one needs to specify the compilers used, the location of the MX API, the prefix of the location of the installation, the program used to spawn processes (rsh/ssh), and various flags required for compilation.
For instance the customized lines in the mpich.make.linux file at IFIN_Bio read:
CC=${CC:-gcc}
CXX=${CXX:-g++}
FC=${FC:-f77}
F90=${F90:-f90}
MX_HOME=${MX_HOME:-/opt/mx}
PREFIX=${PREFIX:-/opt/mpich-mx}
RSHCOMMAND=${RSHCOMMAND:-ssh}
CFLAGS="$CFLAGS -I$MX_HOME/include"
LINKOPT=${LINKOPT:- -Wl,-rpath,$MX_HOME/lib64,-rpath,$MX_HOME/lib
-L$MX_HOME/lib64 -L$MX_HOME/lib/ -lmyriexpress}
OPTFLAGS=${OPTFLAGS:- -O}
If the installation of MPICH-MX fails, one must check the installer script and correct the template corresponding to the current architecture. The structure of the machine file, which specifies the nodes and the number of cores, is similar to the one for the standard MPICH. For example, in the case of the 32 8-core nodes of the IFIN_Bio cluster:
#IFIN_Bio mdqc01.hpc64.nipne.ro:8 mdqc02.hpc64.nipne.ro:8 ... mdqc32.hpc64.nipne.ro:8
Runtime optimization
Regarding the structure of the programs parallelized under MPICH-MX, it should be noted that the MPI header must be included as
#include "/opt/mpich-mx/include/mpi.h"
MPICH-MX fully respects the MPICH syntax, all the names of the MPICH functions are unchanged, and the execution command is mpirun.
MPICH-MX provides a perl script named mpirun.ch_mx, that is used to spawn parallel processes. This can be launched with various parameters in order to improve the performance of the running program, as follows:
mpirun.ch_mx [options] [-np <n>] program_name [flags]
where np denotes the number of processes to be run, and flags are the program’s arguments.
The options allow for chosing between lower latency or higher processing power, depending on the type of program.
If low latency is needed, the right option is
--mx-recv polling
which is the default, as this ensures the highest performance when each processor supports the running of only one process. In this case MPI continually checks the driver ch_mx for the completion of the send/receive events, resulting in a lower latency and a higher processor load.
If the developer is interested in running more than one process on the same processor, as a result of MPI spawning (e.g. GAMESS) or in the case of some multi-threaded applications, then it is recommended that he/she uses the option
--mx-recv blocking
The effect is that MPI will wait for an interrupt from the Myrinet interface, decreasing the usage of the CPU.
In the case when one of the processes unexpectedly dies or exits due to an error, one can avoid the useless continuation of the running of the application by using the option:
--mx-kill <n>
This will ensure the killing of the remaining processes n seconds after the unexpected event, by sending them an ABORT message which is external to the MPI code.
REFERENCES
[myrc] http://www.myricom.com
