Excellent! On Jul 27, 2012, at 4:17 PM, Darius Buntinas wrote:

Another satisfied customer!

-d

Begin forwarded message:

...

From: <[email protected]> Subject: Re: Regarding some more advanced aspects of MPI Date: July 27, 2012 3:34:37 PM CDT To: "Darius Buntinas" <[email protected]>

Hi Darius,

Thanks for the response!

...

I'm not sure how OpenMPI works ...

I came to this conclusion a couple of days ago; after massive frustration in trying to get OpenMPI to recognise me registering error handlers, and not totally dying when I deliberately killed a process, I finally discovered that this behaviour is by design; OpenMPI simply cannot handle one or more processes dying! As such, it's useless to me.

I did, however, install MPICH2 and it works like a dream!

I now have a fault tolerant backend broadcast system, which automatically detects failed nodes and informs the rest of the simulation. Some nice side effects of how I've implemented this are:

1. If any node fails, all the other nodes are automatically made aware of this in the background. The specific manner in which I implement the "broadcast" messages also uncovers automatically any other nodes which have died without anyone otherwise noticing. Therefore, *any* use of the backend communications channel will automatically reveal to all nodes exactly which other nodes have crashed. I think this will be very useful to me in future!

2. All the backend communications should be very scalable, as the broadcasts are split into smaller "block communications", which are sent to the first node in the specified range. This node retransmits the message to the other nodes in that block range. The block messages themselves are subject to re-splitting, down to some specified minimum block size - at which point they are sent directly using point-to-point communications (the actual messages are very small, so can fit into MPI's "eager" send packets). Hopefully, this should scale very well as we're using what amounts to a tree structure to send in parallel. Speaking of which ...

3. The parallel broadcast is itself fault tolerant; there's no single point of failure, and any node can die at any time without stopping the background messages getting to every node. As we define ranges over which the block messages operate, we can easily iterate through the block members from the first member, and pass the message onto the next one which is capable of handling it. The broadcast "tree" for these collective communications is dynamic and fault tolerant, in that sense.

...

-disable-auto-cleanup

That's exactly what I use! :)

Kudos to you guys for MPICH2 - it's very impressive, and I have now been able to implement pretty much exactly what I needed without having to use multicasts over IPv4/6, ibverbs or whatever I feared I may have to use. With it being plain old MPI v2, I should be able to deploy systems using this approach to the Blue Gene machines, clusters with Gemini interconnects etc using just the provided software on the clusters themselves, while also taking automatic advantage of the tuned MPI libraries for each.

I'm really happy with how it's turned out, and the fact that I managed to do so in "pure" MPI. It was a fun technical exercise over a few days!

Cheers,

J.

Awesome! But somehow Open MPI managed to give an impression to all of the HPC community that they are somehow ahead of us in Fault Tolerance. -- Pavan On 07/27/2012 04:17 PM, Darius Buntinas wrote:

Another satisfied customer!

-d

Begin forwarded message:

...

From: <[email protected]> Subject: Re: Regarding some more advanced aspects of MPI Date: July 27, 2012 3:34:37 PM CDT To: "Darius Buntinas" <[email protected]>

Hi Darius,

Thanks for the response!

...

I'm not sure how OpenMPI works ...

I came to this conclusion a couple of days ago; after massive frustration in trying to get OpenMPI to recognise me registering error handlers, and not totally dying when I deliberately killed a process, I finally discovered that this behaviour is by design; OpenMPI simply cannot handle one or more processes dying! As such, it's useless to me.

I did, however, install MPICH2 and it works like a dream!

I now have a fault tolerant backend broadcast system, which automatically detects failed nodes and informs the rest of the simulation. Some nice side effects of how I've implemented this are:

1. If any node fails, all the other nodes are automatically made aware of this in the background. The specific manner in which I implement the "broadcast" messages also uncovers automatically any other nodes which have died without anyone otherwise noticing. Therefore, *any* use of the backend communications channel will automatically reveal to all nodes exactly which other nodes have crashed. I think this will be very useful to me in future!

2. All the backend communications should be very scalable, as the broadcasts are split into smaller "block communications", which are sent to the first node in the specified range. This node retransmits the message to the other nodes in that block range. The block messages themselves are subject to re-splitting, down to some specified minimum block size - at which point they are sent directly using point-to-point communications (the actual messages are very small, so can fit into MPI's "eager" send packets). Hopefully, this should scale very well as we're using what amounts to a tree structure to send in parallel. Speaking of which ...

3. The parallel broadcast is itself fault tolerant; there's no single point of failure, and any node can die at any time without stopping the background messages getting to every node. As we define ranges over which the block messages operate, we can easily iterate through the block members from the first member, and pass the message onto the next one which is capable of handling it. The broadcast "tree" for these collective communications is dynamic and fault tolerant, in that sense.

...

-disable-auto-cleanup

That's exactly what I use! :)

Kudos to you guys for MPICH2 - it's very impressive, and I have now been able to implement pretty much exactly what I needed without having to use multicasts over IPv4/6, ibverbs or whatever I feared I may have to use. With it being plain old MPI v2, I should be able to deploy systems using this approach to the Blue Gene machines, clusters with Gemini interconnects etc using just the provided software on the clusters themselves, while also taking automatic advantage of the tuned MPI libraries for each.

I'm really happy with how it's turned out, and the fact that I managed to do so in "pure" MPI. It was a fun technical exercise over a few days!

Cheers,

J.

-- Pavan Balaji http://www.mcs.anl.gov/~balaji