- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Hydra torque support issues
by Pavan Balaji 07 Mar '13

07 Mar '13

This was already present in 1.5, though not enabled by default. However, there were some bug fixes in 3.0, so using the latest version is the best bet. -- Pavan On 03/07/2013 03:57 PM US Central Time, Dave Goodell wrote: > On Mar 7, 2013, at 3:51 PM CST, Bharath Ramesh <bramesh(a)vt.edu> wrote: > >> I am using mvapich2-1.9a2 which is based of mpich2-1.5. We have >> enabled Torque integration with hydra. We are noticing an issue >> where in Torque is not tracking the resource used by the MPI >> application when they are built with mvapich2. Further >> investigation revealed that mvapich2 hydra process launcher was >> not setting the correct session id to what torque used for >> forking the shell. I am wondering if this is a known issue or >> something that has already been fixed. If it has been fixed, what >> would be the best way to upgrade just hydra without affecting the >> rest of MPI stack. > > I can't speak to whether the issue is known or fixed (Pavan will know). But you can install a different version of Hydra from one of the release tarballs or nightly tarballs: > > http://www.mpich.org/downloads/ > http://www.mpich.org/static/tarballs/3.0.2/hydra-3.0.2.tar.gz > http://www.mpich.org/static/tarballs/nightly/hydra/ > > -Dave > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Hydra torque support issues
by Dave Goodell 07 Mar '13

07 Mar '13

On Mar 7, 2013, at 3:51 PM CST, Bharath Ramesh <bramesh(a)vt.edu> wrote: > I am using mvapich2-1.9a2 which is based of mpich2-1.5. We have > enabled Torque integration with hydra. We are noticing an issue > where in Torque is not tracking the resource used by the MPI > application when they are built with mvapich2. Further > investigation revealed that mvapich2 hydra process launcher was > not setting the correct session id to what torque used for > forking the shell. I am wondering if this is a known issue or > something that has already been fixed. If it has been fixed, what > would be the best way to upgrade just hydra without affecting the > rest of MPI stack. I can't speak to whether the issue is known or fixed (Pavan will know). But you can install a different version of Hydra from one of the release tarballs or nightly tarballs: http://www.mpich.org/downloads/ http://www.mpich.org/static/tarballs/3.0.2/hydra-3.0.2.tar.gz http://www.mpich.org/static/tarballs/nightly/hydra/ -Dave

1 0

Hydra torque support issues
by Bharath Ramesh 07 Mar '13

07 Mar '13

I am using mvapich2-1.9a2 which is based of mpich2-1.5. We have enabled Torque integration with hydra. We are noticing an issue where in Torque is not tracking the resource used by the MPI application when they are built with mvapich2. Further investigation revealed that mvapich2 hydra process launcher was not setting the correct session id to what torque used for forking the shell. I am wondering if this is a known issue or something that has already been fixed. If it has been fixed, what would be the best way to upgrade just hydra without affecting the rest of MPI stack. -- Bharath

1 0

Re: [mpich-discuss] MPI_Comm_Spawn causing zombies of hydra_pmi_proxy
by Pavan Balaji 07 Mar '13

07 Mar '13

I just tried this with the mpich master and it seems to work correctly, and there are no zombie processes (though I reduced the number of iterations to 10000, instead of 200000). This was a problem in mpich once upon a time, but that was a few years ago. Are you using the latest version of mpich (3.0.2)? -- Pavan On 03/07/2013 09:44 AM US Central Time, Silvan Brï¿½ndli wrote: > PS: The attached programs are a simplification of my code. They > reproduce the zombie problem. Waiting for the 32k zombies takes a > while... but I expect the same behaviour as with my original code. > > Am I missing something when finishing the called program? I just use > MPI_Comm_disconnect and MPI_Finalize. > > Best regards > Silvan > > main.cpp > > #include <mpi.h>; > > int main(int argc, char *argv[]) > { > int myrank; > int spawnerror; > int value = 123; > void *buf = &value; > MPI_Comm child_comm; > > if (MPI_Init(&argc,&argv)!=MPI_SUCCESS) > { > printf("MPI_Init failed"); > } > > MPI_Comm_rank(MPI_COMM_WORLD, &myrank); > > char* hiargv[] = {"23",NULL}; > for(int i = 1; i <= 200000; i++) > { > value = i; > printf("Main before spawn %d\n",i); > MPI_Comm_spawn("./hi",hiargv, 1, MPI_INFO_NULL, myrank, > MPI_COMM_SELF, &child_comm, &spawnerror); > MPI_Send(buf, 1, MPI_INTEGER, 0, 1, child_comm); > MPI_Comm_disconnect(&child_comm); > } > > MPI_Finalize(); > return 0; > } > > hi.cpp: > > #include <mpi.h>; > > int main(int argc, char** argv) { > MPI_Comm parent; > MPI_Status status; > int err; > int value = -1; > void* buf= &value; > > if (MPI_Init(&argc,&argv)!=MPI_SUCCESS) > { > printf("MPI_Init failed"); > } > MPI_Comm_get_parent(&parent); > if (parent == MPI_COMM_NULL) printf("No parent!"); > > MPI_Recv(buf, 1, MPI_INTEGER, 0, MPI_ANY_TAG, parent, &status); > MPI_Comm_disconnect(&parent); > err = MPI_Finalize(); > printf("hi finalized %d %d \n",err, value); > return 0; > } > > > > On 07.03.2013 12:38, Silvan Brï¿½ndli wrote: >> Dear all, >> >> again I have a question related to spawning processes. I understand the >> situation as follows: >> >> My program A spawns program B. Program B spawns program C1, C2 ... >> C10000 ... >> Program Cx terminates correctly before Cx+1 is called, however returning >> 1 to mpiexec. To handle this I use the workaround as described in >> http://lists.mpich.org/pipermail/discuss/2013-February/000429.html >> >> Now it looks like with every Spawn a "hydra_pmi_proxy" is started, the >> calling program is mpiexec. When the program Cx is finished this >> "hydra_pmi_proxy" remains as a zombie until the programs A, B and >> mpiexec are finished. When approx. 32k of those "hydra_pmi_proxy" exist >> I get some problems (too many processes or something similar). >> >> What can I do to finish "hydra_pmi_proxy" while my programs A, B and >> mpiexec are still running? >> >> I'm glad about every hint. >> >> Best regards >> Silvan >> > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] MPI_Comm_Spawn causing zombies of hydra_pmi_proxy
by Silvan Brändli 07 Mar '13

07 Mar '13

PS: The attached programs are a simplification of my code. They reproduce the zombie problem. Waiting for the 32k zombies takes a while... but I expect the same behaviour as with my original code. Am I missing something when finishing the called program? I just use MPI_Comm_disconnect and MPI_Finalize. Best regards Silvan main.cpp #include <mpi.h>; int main(int argc, char *argv[]) { int myrank; int spawnerror; int value = 123; void *buf = &value; MPI_Comm child_comm; if (MPI_Init(&argc,&argv)!=MPI_SUCCESS) { printf("MPI_Init failed"); } MPI_Comm_rank(MPI_COMM_WORLD, &myrank); char* hiargv[] = {"23",NULL}; for(int i = 1; i <= 200000; i++) { value = i; printf("Main before spawn %d\n",i); MPI_Comm_spawn("./hi",hiargv, 1, MPI_INFO_NULL, myrank, MPI_COMM_SELF, &child_comm, &spawnerror); MPI_Send(buf, 1, MPI_INTEGER, 0, 1, child_comm); MPI_Comm_disconnect(&child_comm); } MPI_Finalize(); return 0; } hi.cpp: #include <mpi.h>; int main(int argc, char** argv) { MPI_Comm parent; MPI_Status status; int err; int value = -1; void* buf= &value; if (MPI_Init(&argc,&argv)!=MPI_SUCCESS) { printf("MPI_Init failed"); } MPI_Comm_get_parent(&parent); if (parent == MPI_COMM_NULL) printf("No parent!"); MPI_Recv(buf, 1, MPI_INTEGER, 0, MPI_ANY_TAG, parent, &status); MPI_Comm_disconnect(&parent); err = MPI_Finalize(); printf("hi finalized %d %d \n",err, value); return 0; } On 07.03.2013 12:38, Silvan Brï¿½ndli wrote: > Dear all, > > again I have a question related to spawning processes. I understand the > situation as follows: > > My program A spawns program B. Program B spawns program C1, C2 ... > C10000 ... > Program Cx terminates correctly before Cx+1 is called, however returning > 1 to mpiexec. To handle this I use the workaround as described in > http://lists.mpich.org/pipermail/discuss/2013-February/000429.html > > Now it looks like with every Spawn a "hydra_pmi_proxy" is started, the > calling program is mpiexec. When the program Cx is finished this > "hydra_pmi_proxy" remains as a zombie until the programs A, B and > mpiexec are finished. When approx. 32k of those "hydra_pmi_proxy" exist > I get some problems (too many processes or something similar). > > What can I do to finish "hydra_pmi_proxy" while my programs A, B and > mpiexec are still running? > > I'm glad about every hint. > > Best regards > Silvan > -- Dipl.-Ing. Silvan Brï¿½ndli Numerische Strukturanalyse mit Anwendungen in der Schiffstechnik (M-10) Technische Universitï¿½t Hamburg-Harburg Schwarzenbergstraï¿½e 95c 21073 Hamburg Tel. : +49 (0)40 42878 - 6187 Fax. : +49 (0)40 42878 - 6090 e-mail: silvan.braendli(a)tuhh.de www : http://www.tuhh.de/skf 5th GACM Colloquium on Computational Mechanics http://www.tu-harburg.de/gacm2013

1 0

MPI_Comm_Spawn causing zombies of hydra_pmi_proxy
by Silvan Brändli 07 Mar '13

07 Mar '13

Dear all, again I have a question related to spawning processes. I understand the situation as follows: My program A spawns program B. Program B spawns program C1, C2 ... C10000 ... Program Cx terminates correctly before Cx+1 is called, however returning 1 to mpiexec. To handle this I use the workaround as described in http://lists.mpich.org/pipermail/discuss/2013-February/000429.html Now it looks like with every Spawn a "hydra_pmi_proxy" is started, the calling program is mpiexec. When the program Cx is finished this "hydra_pmi_proxy" remains as a zombie until the programs A, B and mpiexec are finished. When approx. 32k of those "hydra_pmi_proxy" exist I get some problems (too many processes or something similar). What can I do to finish "hydra_pmi_proxy" while my programs A, B and mpiexec are still running? I'm glad about every hint. Best regards Silvan -- Dipl.-Ing. Silvan Brï¿½ndli Numerische Strukturanalyse mit Anwendungen in der Schiffstechnik (M-10) Technische Universitï¿½t Hamburg-Harburg Schwarzenbergstraï¿½e 95c 21073 Hamburg Tel. : +49 (0)40 42878 - 6187 Fax. : +49 (0)40 42878 - 6090 e-mail: silvan.braendli(a)tuhh.de www : http://www.tuhh.de/skf 5th GACM Colloquium on Computational Mechanics http://www.tu-harburg.de/gacm2013

1 0

Help with eclipse juno and mpich2
by George Papageorgakis 06 Mar '13

06 Mar '13

Hello people! I am new in mpi. I have been trying to setup eclipse to play with mpich2 in windows 7. I have installed mpich2 and managed to make the eclipse editor identify mpi libs, functions etc but it wont compile correctly. I am using the example of mpi_heat2D and i would like to run it on a ubuntu cluster of my university. I have done so with a kinda slow and weary methodology using WinSCP to copy the .c file from my pc to a cluster pc, compile it and run it via console not being able to see any graphs etc, only the output file with the results. I would like to know what can i do to run and see everything from eclipse, including graphs for optimization etc. I didnt find much help on the internet and i am kinda frustrated. Is what i ask possible? Thanks in advance!

1 0

Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept
by Pavan Balaji 05 Mar '13

05 Mar '13

On 03/05/2013 10:34 AM US Central Time, Jim Dinan wrote: > You should ignore my comments on MPICH FT. My info is clearly > out-of-date. It sounds like what you're looking for should be fully > supported. :) Well, almost :-). Some things could not be done cleanly while staying within MPI-3. For example, when you do a wildcard receive, it will always return an error if any process in the communicator is dead. The MPI Forum is working on fixing this by allowing the user to "opt in" for this wildcard stuff. I'm a few months behind on the MPI-3.1 fault-tolerance proposal, but I believe this is still present. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept
by Jim Dinan 05 Mar '13

05 Mar '13

Hi Mathieu, You should ignore my comments on MPICH FT. My info is clearly out-of-date. It sounds like what you're looking for should be fully supported. :) ~Jim. On 3/5/13 9:42 AM, Matthieu Dorier wrote: > Alright, thanks for the answer (and for the ticket). > Cheers, > > Matthieu > > ----- Mail original ----- >> De: "Jim Dinan" <dinan(a)mcs.anl.gov> >> ï¿½: discuss(a)mpich.org >> Envoyï¿½: Mardi 5 Mars 2013 16:33:31 >> Objet: Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept >> >> Hi Mathieu, >> >> I created an MPI Forum ticket for this: >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/365 >> >> In terms of what is guaranteed by the standard, the behavior is >> undefined. In terms of that MPICH will do, I am not sure, although >> my >> guess is that current MPICH will be unable to continue working after >> such a failure. You may need to do some testing or read the code to >> find out. >> >> Cheers, >> ~Jim. >> >> On 3/4/13 7:56 AM, Matthieu Dorier wrote: >>> Hi, >>> >>> I'm connecting two MPI applications A and B using MPI_Comm_accept >>> in A >>> and MPI_Comm_connect in B. I would like to know a bit more about >>> the >>> behavior in case one application stops (say B): will a >>> communication >>> attempt (e.g. MPI_Send) from a process from A to a process from B >>> crash? >>> return an error? block? >>> Is there a way for application A to notice that B has stopped in >>> order >>> to avoid communicating with it? >>> >>> Thanks. >>> >>> PS: by the way for whom is involved in the MPI forum, an >>> MPI_Comm_iaccept in the MPI3 standard would have been useful. >>> Something >>> to keep in mind for the next version maybe ;) >>> >>> Matthieu Dorier >>> PhD student at ENS Cachan Brittany and IRISA >>> http://people.irisa.fr/Matthieu.Dorier >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept
by Pavan Balaji 05 Mar '13

05 Mar '13

On 03/05/2013 09:33 AM US Central Time, Jim Dinan wrote: > I created an MPI Forum ticket for this: > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/365 Jim: iaccept was a part of Josh's original MPI-3 proposal, but was voted down by the Forum because it's collective over the communicator (though that's true for all nonblocking collectives). I, myself, pointed out several use cases for iaccept (one of which was VOCL, where we have to use a separate thread for comm_accept). It might be worthwhile revisiting it for MPI-3.1 or MPI-4. > In terms of what is guaranteed by the standard, the behavior is > undefined. In terms of that MPICH will do, I am not sure, although my > guess is that current MPICH will be unable to continue working after > such a failure. You may need to do some testing or read the code to > find out. FWIW, the default behavior in mpich is to clean up all processes if something goes wrong. However, you can pass -disable-auto-cleanup to mpiexec to disable this behavior. In this case, if one process dies, the remaining processes are left alone, and will return an error on communication -- you'll obviously need to check the return value of the MPI functions to detect such failures. One small gotcha is the interaction of such failures with MPI-3 nonblocking collectives. Currently, we don't have a way to get these two working together, though that's planned to be fixed in mpich-3.1 to be released in 2014 (preview releases will show up later this year). You can use the environment MPICH_ENABLE_COLL_FT_RET=1 to tell MPICH to not hang, and return an error, if a process fails during a collective operation. But doing this will cause nonblocking collectives to break. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0