July 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Error in MPI_Finalize on a simple ring test over TCP
by Wesley Bland 10 Jul '13

10 Jul '13

Can you send us the smallest chunk of code that still exhibits this error? Wesley On Jul 10, 2013, at 6:54 AM, Thomas Ropars <thomas.ropars(a)epfl.ch> wrote: > Hi all, > > I get the following error when I try to run a simple application implementing a ring (each process sends to rank+1 and receives from rank-1). More precisely, the error occurs during the call to MPI_Finalize(): > > Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 363: sc->pg_is_set > internal ABORT - process 0 > > Does anybody else also noticed the same error? > > Here are all the details about my test: > - The error is generated with mpich-3.0.2 (but I noticed the exact same error with mpich-3.0.4) > - I am using IPoIB for communication between nodes (The same thing happens over Ethernet) > - The problem comes from TCP links. When all processes are on the same node, there is no error. As soon as one process is on a remote node, the failure occurs. > - Note also that the failure does not occur if I run a more complex code (eg, a NAS benchmark). > > Thomas Ropars > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Error in MPI_Finalize on a simple ring test over TCP
by Thomas Ropars 10 Jul '13

10 Jul '13

Hi all, I get the following error when I try to run a simple application implementing a ring (each process sends to rank+1 and receives from rank-1). More precisely, the error occurs during the call to MPI_Finalize(): Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 363: sc->pg_is_set internal ABORT - process 0 Does anybody else also noticed the same error? Here are all the details about my test: - The error is generated with mpich-3.0.2 (but I noticed the exact same error with mpich-3.0.4) - I am using IPoIB for communication between nodes (The same thing happens over Ethernet) - The problem comes from TCP links. When all processes are on the same node, there is no error. As soon as one process is on a remote node, the failure occurs. - Note also that the failure does not occur if I run a more complex code (eg, a NAS benchmark). Thomas Ropars

1 0

MPICH3.0.4 make fails with "No rule to make target..."
by Don Warren 10 Jul '13

10 Jul '13

Hello, As requested in the installation guide, I'm informing this list of a failure to correctly make MPICH3.0.4 on a Win7 system. The specific error encountered is "make[2]: *** No rule to make target `/cygdrive/c/FLASH/mpich-3.0.4/src/mpi/romio/Makefile.am', needed by `/cygdrive/c/FLASH/mpich-3.0.4/src/mpi/romio/Makefile.in'. Stop." I have confirmed that both Makefile.am and Makefile.in exist in the directory listed. I'm attaching the c.txt and the m.txt files. Possibly of interest is that the command "make clean" fails at exactly the same folder, with exactly the same error message as shown in m.txt and above. Any advice you can give would be appreciated. I'm attempting to get FLASH running on my computer, which seems to require MPICH. Regards, Don Warren

1 0

Re: [mpich-discuss] MPI Benchmark Suite
by Rajeev Thakur 10 Jul '13

10 Jul '13

Good write-up. If you want some more, see http://www.nersc.gov/systems/trinity-nersc-8-rfp/draft-nersc-8-trinity-benc… Rajeev On Jul 9, 2013, at 3:03 PM, Jiri Simsa wrote: > The overview is also available here: http://www.cs.cmu.edu/~jsimsa/mpi-macro.pdf > > > On Tue, Jul 9, 2013 at 3:53 PM, Jiri Simsa <jsimsa(a)cs.cmu.edu> wrote: > I have compiled the pointers from Pavan, Rajeev, and Jeff into an overview of MPI workloads (attached). > > Any comments or corrections are welcome. > > --Jiri > > > On Sat, Jul 6, 2013 at 1:38 AM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > See also the Mantevo mini-apps. http://www.mantevo.org/ > > > On Jul 5, 2013, at 11:22 AM, Pavan Balaji wrote: > > > > > On 07/05/2013 10:38 AM, Jiri Simsa wrote: > >> Could anyone point me in the direction of programs that are > >> representative of MPI programs? I am looking for something like PARSEC > >> (http://parsec.cs.princeton.edu/) for MPI. In other words, I am > >> interested in macro-bechnmarks not micro-benchmarks. Thank you. > > > > You could try the Sequoia suite. There's also the NAS parallel benchmarks and Graph500 benchmarks. > > > > -- Pavan > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Fwd: MPICH2 installation and running problem
by Sarath Samarasekara 09 Jul '13

09 Jul '13

Hi, I am trying to install MPICH2 in my PC (running widows 7, 64 bit). I have tried with the following versions, but I couldn't do a successful installation. MPICH2-1.0.6p1-win32-ia32 MPICH2-1.4.1p1-win-ia32 MPICH2-1.4.1p1-win-x86-64 MPICH2-1.0-1-win32-ia32 Administrator command prompt used. I have added C:\Program Files\MPICH2\bin" to system Path Control panel- Administrative tools-Services shows MPICH2 but it doesn't show SMPD. SMPD -status just hangs at the command prompt. Following is an output of SMPD -status -d for you information. What am I missing?. How do I get this going? Thank you for your advice in advance. Regards, Sam C:\>smpd -status -d [-1:3564]..\smpd_get_opt_int [-1:3564]../smpd_get_opt_int [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt_int [-1:3564]../smpd_get_opt_int [-1:3564]..\smpd_get_opt_int [-1:3564]../smpd_get_opt_int [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt_string [-1:3564]../smpd_get_opt_string [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_hostname [-1:3564]../smpd_get_hostname [-1:3564]..\smpd_get_opt_string [-1:3564]../smpd_get_opt_string [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt_string [-1:3564]../smpd_get_opt_string [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt_string [-1:3564]../smpd_get_opt_string [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt_string [-1:3564]../smpd_get_opt_string [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt_string [-1:3564]../smpd_get_opt_string [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_get_opt_string [-1:3564]../smpd_get_opt_string [-1:3564]..\smpd_get_opt_string [-1:3564]../smpd_get_opt_string [-1:3564]..\smpd_get_opt [-1:3564]../smpd_get_opt [-1:3564]..\smpd_do_console [-1:3564]...\smpd_get_smpd_data [-1:3564]....\smpd_get_smpd_data_from_environment [-1:3564]..../smpd_get_smpd_data_from_environment [-1:3564].../smpd_get_smpd_data [-1:3564]...\SMPDU_Sock_create_set no smpd running on PC [00:3564].../smpd_do_console [00:3564]...\smpd_exit [00:3564]....\smpd_kill_all_processes [00:3564]..../smpd_kill_all_processes [00:3564]....\smpd_finalize_drive_maps [00:3564]..../smpd_finalize_drive_maps C:\>

1 0

Re: [mpich-discuss] Segmentation fault (signal 11) and Exit code 139
by Jeff Hammond 09 Jul '13

09 Jul '13

You compiled with Intel Fortran? That compiler puts static arrays on the stack, unlike GNU Fortran, which puts them on the heap. You can either use ulimit to make the stack huge or you can look up the compiler option for Intel that puts these arrays on the heap. Only ancient, poorly designed Fortran 77 codes have this problem, by the way. Jeff On Tue, Jul 9, 2013 at 8:49 PM, Zheng Li <seraphzl(a)gmail.com> wrote: > ulimit -s 262140 > By setting the stack size to 256M, this issue was solved. > > Lee > > > 2013/7/9 Jeff Hammond <jeff.science(a)gmail.com> >> >> Not an MPICH problem. Contact WRF support. >> >> Jeff >> >> Sent from my iPhone >> >> On Jul 9, 2013, at 2:09 AM, Zheng Li <seraphzl(a)gmail.com> wrote: >> >> Dear All, >> >> I tried to run the default January 2000 Case (the East Coast Winter Storm >> of January 24-25, 2000) in WRF v3.5, and failed when running wrf.exe. The >> error is as follows: >> >> [sl@sl0 em_real]$ mpirun -np 4 ./wrf.exe >> starting wrf task 0 of 4 >> starting wrf task 1 of 4 >> starting wrf task 2 of 4 >> starting wrf task 3 of 4 >> >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = EXIT CODE: 139 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >> =================================================================================== >> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault >> (signal 11) >> This typically refers to a problem with your application. >> Please see the FAQ page for debugging suggestions >> >> Does someone have idea of resolving this issue? Thanks in advance. >> >> Lee >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Dr. Zheng Li > Research Assistant > State Key Laboratory of Environmental Aquatic Chemistry > Research Center for Eco-Environmental Sciences > Chinese Academy of Sciences > P. O. Box 2871, Beijing, 100085 China > seraphzl(a)gmail.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] Segmentation fault (signal 11) and Exit code 139
by Zheng Li 09 Jul '13

09 Jul '13

ulimit -s 262140 By setting the stack size to 256M, this issue was solved. Lee 2013/7/9 Jeff Hammond <jeff.science(a)gmail.com> > Not an MPICH problem. Contact WRF support. > > Jeff > > Sent from my iPhone > > On Jul 9, 2013, at 2:09 AM, Zheng Li <seraphzl(a)gmail.com> wrote: > > Dear All, > > I tried to run the default January 2000 Case (the East Coast Winter Storm > of January 24-25, 2000) in WRF v3.5, and failed when running wrf.exe. The > error is as follows: > > [sl@sl0 em_real]$ mpirun -np 4 ./wrf.exe > starting wrf task 0 of 4 > starting wrf task 1 of 4 > starting wrf task 2 of 4 > starting wrf task 3 of 4 > > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = EXIT CODE: 139 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault > (signal 11) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions > > Does someone have idea of resolving this issue? Thanks in advance. > > Lee > <seraphzl(a)gmail.com> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dr. Zheng Li Research Assistant State Key Laboratory of Environmental Aquatic Chemistry Research Center for Eco-Environmental Sciences Chinese Academy of Sciences P. O. Box 2871, Beijing, 100085 China seraphzl(a)gmail.com

1 0

Re: [mpich-discuss] MPI Benchmark Suite
by Pavan Balaji 09 Jul '13

09 Jul '13

Nice collection. Thanks for putting it together. -- Pavan On 07/09/2013 02:53 PM, Jiri Simsa wrote: > I have compiled the pointers from Pavan, Rajeev, and Jeff into an > overview of MPI workloads (attached). > > Any comments or corrections are welcome. > > --Jiri > > > On Sat, Jul 6, 2013 at 1:38 AM, Rajeev Thakur <thakur(a)mcs.anl.gov > <mailto:[email protected]>> wrote: > > See also the Mantevo mini-apps. http://www.mantevo.org/ > > > On Jul 5, 2013, at 11:22 AM, Pavan Balaji wrote: > > > > > On 07/05/2013 10:38 AM, Jiri Simsa wrote: > >> Could anyone point me in the direction of programs that are > >> representative of MPI programs? I am looking for something like > PARSEC > >> (http://parsec.cs.princeton.edu/) for MPI. In other words, I am > >> interested in macro-bechnmarks not micro-benchmarks. Thank you. > > > > You could try the Sequoia suite. There's also the NAS parallel > benchmarks and Graph500 benchmarks. > > > > -- Pavan > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Support for MIC in MPICH 3.0.4
by Maciej.Golebiewski＠csiro.au 09 Jul '13

09 Jul '13

Hi Jeff, > Does your code actually scale to 240 MPI ranks per card? What is > the performance relative to peak or a fully-populated MPI-only run > on e.g SNB? I don't expect it to scale up to that many ranks, mentioning 240 ranks/card in my previous email was just to demonstrate that I was able to start that many ranks with Intel MPI. Performance peaks out at 60 ranks on MIC and is still about 4.5 times slower than 12 ranks on a dual slot system with 6 core X5670 Xeons (on which it scales up to 36 ranks over 3 nodes, the grid is only about 15K points which is probably what prevents further scaling). While MIC cores are slower that Xeon, 4.5 times slower with 5 times as many ranks was quite dissapointing. My first hunch was that perhaps Intel MPI does not handle well many immediate send/recv requests at a time (according to vampir traces node leader is always ready for workers to talk to them, but the workers take long to complete either receive of input or send of results), so I wanted to compare the perfrmance using another MPI implementation. After reading your comment about ring interconenct now I consider the possibility that instead I'm hitting the limit of the interconnect. > You're going to have to convert to MPI+Threads eventually so you That's one of these things that are obvious from a technical point of view but are not going to happen any time soon due to various non-technical impediments. > If you don't like OpenMP, you could consider Pthreads or TBB I've done my fair share of Pthreads programming in the past, and it's an evil thing to recommend it over OpenMP for a scientific application actively developed by people who are not expert programmers. Very, very evil. ;) > OpenMP, it's because you have an F77 code with common blocks that > cannot be made thread-safe for all the beer in Australia. Close enough, but not that bad. F90 with data in multiple modules, the main loop time loop could be nicely task-parallelized, the main effort would be to refactor the code so that no subroutine is accessing data through modules, only through the arguments. And then to forever keep policing the scientists to stick to this rule. :) Again, the technical side is not the problem, it's the external circumstances. Anyway, thanks for your comments. Replying to them helped me tidy my thoughts and get a better idea how to present my results and conclusions to the user. Cheers, Maciej

1 0

Re: [mpich-discuss] MPI Benchmark Suite
by Jiri Simsa 09 Jul '13

09 Jul '13

The overview is also available here: http://www.cs.cmu.edu/~jsimsa/mpi-macro.pdf On Tue, Jul 9, 2013 at 3:53 PM, Jiri Simsa <jsimsa(a)cs.cmu.edu> wrote: > I have compiled the pointers from Pavan, Rajeev, and Jeff into an overview > of MPI workloads (attached). > > Any comments or corrections are welcome. > > --Jiri > > > On Sat, Jul 6, 2013 at 1:38 AM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > >> See also the Mantevo mini-apps. http://www.mantevo.org/ >> >> >> On Jul 5, 2013, at 11:22 AM, Pavan Balaji wrote: >> >> > >> > On 07/05/2013 10:38 AM, Jiri Simsa wrote: >> >> Could anyone point me in the direction of programs that are >> >> representative of MPI programs? I am looking for something like PARSEC >> >> (http://parsec.cs.princeton.edu/) for MPI. In other words, I am >> >> interested in macro-bechnmarks not micro-benchmarks. Thank you. >> > >> > You could try the Sequoia suite. There's also the NAS parallel >> benchmarks and Graph500 benchmarks. >> > >> > -- Pavan >> > >> > -- >> > Pavan Balaji >> > http://www.mcs.anl.gov/~balaji >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> >

1 0