Michael,

Good question.  Since this is the MPICH forum, I ran some test cases on a CRAY, and a 64bit PC: 

For the CRAY using the cray-mpich/6.3.1 module. The code simply initializes MPI and returns its rank.  The code (exit_test.C) follows:

#include <iostream>
#include <cstdlib>
#include <mpi.h>

int main(void) {
    int rank;
    MPI_Init(0,0);
    MPI_Comm_rank( MPI_COMM_WORLD, &rank);
    if( rank == 0 ) {
        int size;
        MPI_Comm_size( MPI_COMM_WORLD, &size);
        std::cout << "Rank:" << rank << ", size:" << size << std::endl;
    }
    MPI_Finalize();
    return rank;
}

I compiled with

CC exit_test.C -o exit_test

and executed with the following (PBS_INTERACTIVE)

aprun -n 7 ./exit_test

I was surprised with the result.  I ran it probably a dozen times.  Here are a few runs ...

xxx@batch2:~/src/MPI/tmp> aprun -n 7 ./exit_test
Rank:0, size:7
Application 13573540 exit codes: 6
Application 13573540 resources: utime ~0s, stime ~1s, Rss ~39164, inblocks ~10234, outblocks ~23470
xxx@batch2:~/src/MPI/tmp> echo $?
6
xxx@batch2:~/src/MPI/tmp> aprun -n 7 ./exit_test
Rank:0, size:7
Application 13573552 exit codes: 4
Application 13573552 resources: utime ~0s, stime ~1s, Rss ~39164, inblocks ~10234, outblocks ~23470
xxx@batch2:~/src/MPI/tmp> echo $?
4
xxx@batch2:~/src/MPI/tmp> aprun -n 7 ./exit_test
Rank:0, size:7
Application 13573564 exit codes: 1
Application 13573564 resources: utime ~0s, stime ~1s, Rss ~39172, inblocks ~10234, outblocks ~23470
xxx@batch2:~/src/MPI/tmp> echo $?
1
xxx@batch2:~/src/MPI/tmp> aprun -n 7 ./exit_test
Rank:0, size:7
Application 13573589 exit codes: 5
Application 13573589 resources: utime ~0s, stime ~1s, Rss ~39188, inblocks ~10234, outblocks ~23470
xxx@batch2:~/src/MPI/tmp> echo $?
5

I say surprised because the man page taken from http://www.mpich.org/static/docs/v3.1/www1/mpiexec.html
reports the following ...

Return Status

mpiexec returns the maximum of the exit status values of all of the processes created by mpiexec.


But the fine Cray folk have probably mucked with the MPI executive, so the results running on the XC30 might not agree with the MPICH man page.

I also ran this test on a PC running UBUNTU 12.04, with source built MPICH 3.1.2.  Here are a few of those results

Compile line: mpicc exit_test.C -o exit_test

Here are results from this test.  Note than I am varying the processor count here.

xxx@Cue ~/tmp/MPI $ mpiexec -np 9 ./exit_test
Rank:0, size:9

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 19062 RUNNING AT Cue
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
xxx@Cue ~/tmp/MPI $ echo $?
15
xxx@Cue ~/tmp/MPI $ mpiexec -np 8 ./exit_test
Rank:0, size:8

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 19130 RUNNING AT Cue
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
xxx@Cue ~/tmp/MPI $ echo $?
7
xxx@Cue ~/tmp/MPI $ mpiexec -np 7 ./exit_test
Rank:0, size:7

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 19142 RUNNING AT Cue
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
xxx@Cue ~/tmp/MPI $ echo $?
7
xxx@Cue ~/tmp/MPI $ mpiexec -np 3 ./exit_test
Rank:0, size:3

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 19158 RUNNING AT Cue
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
xxx@Cue ~/tmp/MPI $ echo $?
3

So unlike the test on the CRAY, the results on the PC are consistent, but do not agree with the man page. 

Michael to address your question, it is certainly implementation dependent.  There are any number of ways to report the exit status. MPICH docs suggest taking the maximum value.  Another approach is to use the exit status of the root process.

As a side note, MPICH acknowledges that a non-zero return value results in a bad termination.  If I run the later case with np=1, I get

xxx@Cue ~/tmp/MPI $ mpiexec -np 1 ./exit_test
Rank:0, size:1
xxx@Cue ~/tmp/MPI $ echo $?
0

which is what I would expect. --Mike

On 04/09/2015 02:27 PM, Michael Raymond wrote: