Adapting code to memory architecture
From HP-SEE Wiki
Contents |
Symmetric multiprocessing
SMP on BG/P
Section contributed by IICT-BAS
Each Blue Gene/P Compute chip contains four PowerPC 450 processor cores, running at 850 MHz. The cores are cache coherent and the chip can operate as a 4-way symmetric multiprocessor (SMP). The memory subsystem on the chip consists of small private L2 caches, a central shared 8 MB L3 cache, and dual DDR2 memory controllers. The bottleneck in the scalability of SMP using crossbar switches is the bandwidth and power consumption of the interconnect among the various processors, the memory, and the disk arrays. For SMP Model in BlueGene/P, each node performs a copy of the program and it can run 1-4 threads, each running of the processor within the node. One should be aware that on the Blue Gene there is no virtual memory. Memory is not divided between CPU cores and thus SMP programs on one CPU are possible, however at most 4 threads can be run. Thus it is more logical for HPC usage to use a hybrid OpenMP - MPI computation model. In such a model there is additional advantage in that the number of MPI processes is less than a pure MPI program would require and thus communication is simplified and less communication buffers are required. A Blue Gene node may be run in three modes - SMP,DUAL,VN. In SMP mode there is one MPI process per node, 4GB memory, up to 4 threads. In DUAL mode there are two MPI processes per node, 2GB memory per process, up to two threads. In virtual-node mode (VN) there are four MPI processes per node, 1GB limit per process.
ccNUMA
Section contributed by NIIFI
Non-Uniform Memory Access (NUMA) is a computer memory design used in Multiprocessing, where the memory access time depends on the memory location relative to a processor. Under NUMA, a processor can access its own local memory faster than non-local memory, that is, memory local to another processor or memory shared between processors. Nearly all CPU architectures use a small amount of very fast non-shared memory known as cache to exploit locality of reference in memory accesses. With NUMA, maintaining cache coherence across shared memory has a significant overhead. NUMA computers are using special-purpose hardware to maintain cache coherence, and thus class as "cache-coherent NUMA", or ccNUMA. Typically, this takes place by using inter-processor communication between cache controllers to keep a consistent memory image when more than one cache stores the same memory location. For this reason, ccNUMA may perform poorly when multiple processors attempt to access the same memory area in rapid succession. Because the Linux operating system has a tendency to migrate processes, the importance of using a placement tool becomes more apparent.
Useful placement tools on the SGI UV machine
dplace
You can use the dplace command to bind a related set of processes to specific CPUs or nodes to prevent process migration. By default, memory is allocated to a process on the node on which the process is executing. If a process moves from node to node while it running, a higher percentage of memory references are made to remote nodes. Because remote accesses typically have higher access times, process performance can be diminished. CPU instruction pipelines also have to be reloaded.
Using the dplace command with MPI Programs:
mpirun -np 12 /usr/bin/dplace -s1 -c 24-35 ./connectivity
Using dplace command with OpenMP Programs:
export OMP_NUM_THREADS 6 /usr/bin/dplace -x6 –c 24-29 ./program</nowiki>
numactl
It runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for command and inherited by all of its children. The processes will be bind to specific memory nodes in this case. Example for numactl:
/usr/bin/numactl –m 5 ./program
The program can access only the number 5 memory node.
cpuset
The cpuset file system is a pseudo-file-system interface to the kernel cpuset mechanism, which is used to control the processor placement and memory placement of processes. It is commonly mounted at /dev/cpuset. Automatic cpuset allocation has been configured on the Pecs UV [pecs] machine which is integrated to the Sun Grid Engine (SGE) scheduler. If a new job is starting then the SGE create a new cpuset for the job which contain only the asked CPUs and their local memory nodes. Other jobs cannot access these.
RESOURCES
[pecs] http://wiki.hp-see.eu/index.php/Resource_centre_Pecs_SC Pecs UV
Distributed memory
HPC clusters
Section contributed by IPB
BG/P
Section contributed by IICT-BAS
As an example application that is adapting the code to memory architecture on IBM BlueGene/P we consider the Gromacs software package, which includes Global Array Toolkit library. Global Array Toolkit is deployed and can be used by user codes. It is an efficient platform-independent API, which provides the so-called Global Array abstraction. This is an array, whose storage is distributed in the memory of all MPI processes, but from the application point of view is accessed uniformly, as if it were in the local memory. This abstracts and hides the distributed nature of the memory of the host machine and one can create arrays as large as needed, as long as they fit in the total combined memory of all computing nodes. The Global array toolkit has additional libraries, including a Memory Allocator (MA), Aggregate Remote Memory Copy Interface (ARMCI), and functionality for out-of-core storage of arrays (ChemIO). Shared memory is used to store global array data and is allocated by the Global Arrays run-time system called ARMCI. ARMCI uses shared memory to optimize performance and avoid explicit interprocessor communication within a single shared memory system or an SMP node. ARMCI allocates shared memory from the operating system in large segments and then manages memory in each segment in response to the GA allocation and deallocation calls. Each segment can hold data in many small global arrays. ARMCI does not return shared memory segments to the operating system until the program terminates. Another application that can be compiled with ARMCI support is GAMESS.
