Hello, I've patched the hydra process manager (mpich release 3.1), to support my case (see attachment). It's a bit ugly (I have not prefixed the sort and compare function names as these are local to the alloc.c file) but it works for me and makes the resource allocation for consecutive MPI_COMM_SPAWNs more fair. All I had to patch was 'utils/alloc/alloc.c' in the hydra source-code. New procedure for allocating proxy's and assigning executables is now: 1. make proxies (NEW: make a proxy for every node, do not stop when number of processes is reached) 2 (NEW). sort the proxies by their nodes active number of processes (decreasing, the first proxy will have the least number of processes) 3. allocate executables to the proxies Should I create a bug report for this? (I can't find a login button on the trac website) Kind regards, Arjen On 12-06-15 11:23, Arjen van Elteren wrote:

Hello,

I'm working with an application that invokes multiple mpi_comm_spawn calls.

I'm using mpiexec on a cluster without a resource manager or job queue, so plain ssh and fork calls and everything is managed by mpich.

It looks like mpiexec (both hydra and mpd) re-use the hostfile from the start (and do not look at already allocated resources/used nodes).

For example I have a hostfile like this:

node01:1 node02:1 node03:1

When I run a call like this:

MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, MPI_INFO_NULL, 2, MPI_COMM_SELF, &worker, MPI_ERRCODES_IGNORE);

I get an allocation like this:

node process --------------- ------------------ node01 manager node02 worker 1 node03 worker 2

Which is what I expected.

But when I instead do 2 calls like this (i.e. each worker has one process, but there are 2 workers):

MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, MPI_INFO_NULL, 1, MPI_COMM_SELF, &worker, MPI_ERRCODES_IGNORE);

MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, MPI_INFO_NULL, 1, MPI_COMM_SELF, &worker, MPI_ERRCODES_IGNORE);

I get an allocation like this (both hydra and mpd):

node process --------------- ------------------ node01 manager node02 worker 1 + worker 2 node03

Which is not what I expected at all!

In fact, when I do this for a more complex example, I conclude that in MPI_Comm_spawn the hostfile is simply re-interpreted for every spawn and previous allocations in the same application are not accounted for.

I know I could set hostname in the MPI_Comm_spawn call, but then I'm moving deployment information into my application (and I don't want to recompile or add a commandline argument for something that should be handled by mpiexec)

Is there an option or easy fix for this problem? (I looked at the code of hydra, but I'm unsure how the different proxy's and processes divide this spawning work between them (I could not easily detect one "grand master" that does the allocation...)

Kind regards,

Arjen

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss