January 2017 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] [MPICH users] Problems using MPICH instead of OpenMPI
by Fischer 31 Jan '17

31 Jan '17

Hi Ken, thanks a lot for your help! I used the correct mpiexec, so that wasn´t the problem. I do not understand why, but I just changed my hostfile using the IP addresses of the machines directly instead of the actual hostname and that worked. Do you know if this is the general case for MPICH or can this be caused by some network settings or installation settings? I am confused since using the hostname for OpenMPI worked just fine. Thanks for your fast reply and help! Lisa Am 30.01.2017 um 16:27 schrieb Kenneth Raffenetti: > One thing to be careful of is that you are not mixing an mpiexec from > openmpi with an application linked with mpich. This can cause issues > with processes being able to connect to each other, which is likely > what is happening in your runs. > > Please double check that you are using the mpiexec from mpich. > > Ken > > On 01/28/2017 04:37 AM, lisa.fischer(a)zib.de wrote: >> Dear all, >> >> I want to use MPICH instead of OpenMPI and have some difficulties. >> Using OpenMPI worked fine. >> >> I run my program on two different machines and set the >> MPICH_PORT_RANGE to >> that of OpenMPI, but it seems like my program is running into a >> deadlock. >> Sometimes I do not get any error message and sometimes I get the >> following: >> >> Assertion failed in file src/mpid/ch3/src/ch3u_handle_connection.c at >> line >> 325: vc->state == MPIDI_VC_STATE_ACTIVE >> internal ABORT - process 0 >> >> Do I need to change the port range of MPICH? Is there a function like >> ompi_info to figure out the port range for MPICH as well? >> Or do you think there is another problem? Do I need to set other >> environment variables as well? >> >> The ssh connection on both machines works. I also used a test program >> using only send and receive calls which led to the same problem. >> >> I would be very happy for any hints. >> >> Thank you and have a good weekend! >> Lisa >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Freundliche Grüße / Kind regards, Lisa Fischer Student Assistant Computational Medicine Numerical Mathematics Mathematics for Life and Materials Sciences Konrad-Zuse-Zentrum für Informationstechnik Berlin Takustraße 7 14195 Berlin-Dahlem lisa.fischer(a)zib.de _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [MPICH users] Problems using MPICH instead of OpenMPI
by Kenneth Raffenetti 30 Jan '17

30 Jan '17

One thing to be careful of is that you are not mixing an mpiexec from openmpi with an application linked with mpich. This can cause issues with processes being able to connect to each other, which is likely what is happening in your runs. Please double check that you are using the mpiexec from mpich. Ken On 01/28/2017 04:37 AM, lisa.fischer(a)zib.de wrote: > Dear all, > > I want to use MPICH instead of OpenMPI and have some difficulties. > Using OpenMPI worked fine. > > I run my program on two different machines and set the MPICH_PORT_RANGE to > that of OpenMPI, but it seems like my program is running into a deadlock. > Sometimes I do not get any error message and sometimes I get the following: > > Assertion failed in file src/mpid/ch3/src/ch3u_handle_connection.c at line > 325: vc->state == MPIDI_VC_STATE_ACTIVE > internal ABORT - process 0 > > Do I need to change the port range of MPICH? Is there a function like > ompi_info to figure out the port range for MPICH as well? > Or do you think there is another problem? Do I need to set other > environment variables as well? > > The ssh connection on both machines works. I also used a test program > using only send and receive calls which led to the same problem. > > I would be very happy for any hints. > > Thank you and have a good weekend! > Lisa > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

[MPICH users] Problems using MPICH instead of OpenMPI
by lisa.fischer＠zib.de 28 Jan '17

28 Jan '17

Dear all, I want to use MPICH instead of OpenMPI and have some difficulties. Using OpenMPI worked fine. I run my program on two different machines and set the MPICH_PORT_RANGE to that of OpenMPI, but it seems like my program is running into a deadlock. Sometimes I do not get any error message and sometimes I get the following: Assertion failed in file src/mpid/ch3/src/ch3u_handle_connection.c at line 325: vc->state == MPIDI_VC_STATE_ACTIVE internal ABORT - process 0 Do I need to change the port range of MPICH? Is there a function like ompi_info to figure out the port range for MPICH as well? Or do you think there is another problem? Do I need to set other environment variables as well? The ssh connection on both machines works. I also used a test program using only send and receive calls which led to the same problem. I would be very happy for any hints. Thank you and have a good weekend! Lisa _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Hybrid HPC system
by Min Si 24 Jan '17

24 Jan '17

It seems some thing wrong with the process group id. Can you try the execution again with MPICH debug message ? You can enable debug as below: - configure MPICH again with --enable-g=all, then make && make install - before executing: mkdir -p log/ export MPICHD_DBG_PG=yes export MPICH_DBG_FILENAME="log/dbg-%d.log" export MPICH_DBG_CLASS=ALL export MPICH_DBG_LEVEL=VERBOSE Then can you send me the output and the log files in log/ ? Min On 1/23/17 10:46 AM, Doha Ehab wrote: > Hi Min, > I have attached the two Config.log. and here is the code > > #include <stdio.h> > #include <mpi.h> > > int main (argc, argv) > int argc; > char *argv[]; > { > > int i=0; > MPI_Init (&argc, &argv);/* starts MPI */ > // Find out rank, size > int world_rank; > MPI_Comm_rank(MPI_COMM_WORLD, &world_rank); > int world_size; > MPI_Comm_size(MPI_COMM_WORLD, &world_size); > > int number; > if (world_rank == 0) { > > number = -1; > for( i=1; i < world_size; i++){ > > MPI_Send(&number, 1, MPI_INT, i, 0, MPI_COMM_WORLD); > } > } > else { > MPI_Recv(&number, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, > MPI_STATUS_IGNORE); > printf("Process %d received number %d from process > 0\n",world_rank, number); > } > MPI_Finalize(); > > return 0; > } > > Regards, > Doha > > On Sun, Jan 22, 2017 at 10:47 PM, Min Si <msi(a)anl.gov > <mailto:[email protected]>> wrote: > > Hi Doha, > > Can you please send us the config.log file for each MPICH build > and your helloworld source doe ? The config.log file should be > under your MPICH build directory where you executed ./configure. > > Min > > On 1/21/17 4:53 AM, Doha Ehab wrote: >> I have tried what you mentioned in the previous E-mail. >> >> 1- I have build MPICH for CPU node and ARM node. >> 2- Uploaded the binaries on same path on the 2 nodes. >> 3- Compiled helloWorld (it sends a number from process zero to >> all other processes ) for both nodes. Then tried mpiexec -np 2 -f >> <hostfile with mic hostnames>./helloworld >> >> I got this error >> Fatal error in MPI_Recv: Other MPI error, error stack: >> MPI_Recv(200)................................: >> MPI_Recv(buf=0xbe9460d0, count=1, MPI_INT, src=0, tag=0, >> MPI_COMM_WORLD, status=0x1) failed >> MPIDI_CH3i_Progress_wait(242)................: an error occurred >> while handling an event returned by MPIDU_Sock_Wait() >> MPIDI_CH3I_Progress_handle_sock_event(554)...: >> MPIDI_CH3_Sockconn_handle_connopen_event(899): unable to find the >> process group structure with id <> >> >> Regards, >> Doha >> >> >> On Wed, Nov 16, 2016 at 6:38 PM, Min Si <msi(a)anl.gov >> <mailto:[email protected]>> wrote: >> >> I guess you might need to put all the MPICH binaries (e.g., >> hydra_pmi_proxy) to the same path on each node. I have >> executed MPICH on Intel MIC chips from the host CPU node >> where OS are different. The thing I did was: >> 1. build MPICH for both CPU node and MIC on the CPU node (you >> have done this step). >> 2. upload the MIC binaries to the same path on MIC chip as on >> the CPU node >> For example: >> - on CPU node : /tmp/mpich/install/bin holds the CPU version >> - on MIC : /tmp/mpich/install/bin holds the MIC >> version >> 3. compile helloworld.c with the MIC version mpicc >> 4. execute on CPU node: mpiexe -np 2 -f <hostfile with mic >> hostnames>./helloworld >> >> I think you should be able to follow step 2, but since your >> helloworld binary is also built with different OS, you might >> want to put it also into the same path on two nodes similar >> as we do for MPICH binaries. >> >> Min >> >> >> On 11/16/16 8:29 AM, Kenneth Raffenetti wrote: >> >> Have you disabled any and all firewalls on both nodes? It >> sounds like they are unable to communicate in initialization. >> >> Ken >> >> On 11/16/2016 07:34 AM, Doha Ehab wrote: >> >> Yes, I built MPICH-3 on both systems and I tried the >> code on each node >> separately and it worked, I tried each node with >> other nodes that has >> the same operating system and it worked as well. >> When I try the code on the 2 nodes that have >> different operating systems >> no result or error message appear. >> >> Regards >> Doha >> >> On Mon, Nov 14, 2016 at 6:25 PM, Kenneth Raffenetti >> <raffenet(a)mcs.anl.gov <mailto:[email protected]> >> <mailto:[email protected] >> <mailto:[email protected]>>> wrote: >> >> It may be possible to run in such a setup, but it >> would not be >> recommended. Did you build MPICH on both systems >> you are trying to >> run on? What exactly happened when the code >> didn't work? >> >> Ken >> >> >> On 11/13/2016 12:36 AM, Doha Ehab wrote: >> >> Hello, >> I tried to run a parallel (Hello World) C >> code on a cluster >> that has 2 >> nodes, the nodes have different operating >> system so the code did not >> work and no results were printed. >> How to make such a cluster work? is there is >> extra steps that >> should be >> done? >> >> Regards, >> Doha >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> <mailto:[email protected]> >> <mailto:[email protected] >> <mailto:[email protected]>> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> <https://lists.mpich.org/mailman/listinfo/discuss> >> >> <https://lists.mpich.org/mailman/listinfo/discuss >> <https://lists.mpich.org/mailman/listinfo/discuss>> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> <https://lists.mpich.org/mailman/listinfo/discuss> >> <https://lists.mpich.org/mailman/listinfo/discuss >> <https://lists.mpich.org/mailman/listinfo/discuss>> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> <https://lists.mpich.org/mailman/listinfo/discuss> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> <https://lists.mpich.org/mailman/listinfo/discuss> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> <https://lists.mpich.org/mailman/listinfo/discuss> >> >> >> >> >> _______________________________________________ >> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> <https://lists.mpich.org/mailman/listinfo/discuss> > _______________________________________________ discuss mailing > list discuss(a)mpich.org <mailto:[email protected]> To manage > subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Hybrid HPC system
by Doha Ehab 23 Jan '17

23 Jan '17

Hi Min, I have attached the two Config.log. and here is the code #include <stdio.h> #include <mpi.h> int main (argc, argv) int argc; char *argv[]; { int i=0; MPI_Init (&argc, &argv); /* starts MPI */ // Find out rank, size int world_rank; MPI_Comm_rank(MPI_COMM_WORLD, &world_rank); int world_size; MPI_Comm_size(MPI_COMM_WORLD, &world_size); int number; if (world_rank == 0) { number = -1; for( i=1; i < world_size; i++){ MPI_Send(&number, 1, MPI_INT, i, 0, MPI_COMM_WORLD); } } else { MPI_Recv(&number, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Process %d received number %d from process 0\n",world_rank, number); } MPI_Finalize(); return 0; } Regards, Doha On Sun, Jan 22, 2017 at 10:47 PM, Min Si <msi(a)anl.gov> wrote: > Hi Doha, > > Can you please send us the config.log file for each MPICH build and your > helloworld source doe ? The config.log file should be under your MPICH > build directory where you executed ./configure. > > Min > > On 1/21/17 4:53 AM, Doha Ehab wrote: > > I have tried what you mentioned in the previous E-mail. > > 1- I have build MPICH for CPU node and ARM node. > 2- Uploaded the binaries on same path on the 2 nodes. > 3- Compiled helloWorld (it sends a number from process zero to all other > processes ) for both nodes. Then tried mpiexec -np 2 -f <hostfile with > mic hostnames>./helloworld > > I got this error > Fatal error in MPI_Recv: Other MPI error, error stack: > MPI_Recv(200)................................: MPI_Recv(buf=0xbe9460d0, > count=1, MPI_INT, src=0, tag=0, MPI_COMM_WORLD, status=0x1) failed > MPIDI_CH3i_Progress_wait(242)................: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > MPIDI_CH3I_Progress_handle_sock_event(554)...: > MPIDI_CH3_Sockconn_handle_connopen_event(899): unable to find the process > group structure with id <> > > Regards, > Doha > > > On Wed, Nov 16, 2016 at 6:38 PM, Min Si <msi(a)anl.gov> wrote: > >> I guess you might need to put all the MPICH binaries (e.g., >> hydra_pmi_proxy) to the same path on each node. I have executed MPICH on >> Intel MIC chips from the host CPU node where OS are different. The thing I >> did was: >> 1. build MPICH for both CPU node and MIC on the CPU node (you have done >> this step). >> 2. upload the MIC binaries to the same path on MIC chip as on the CPU node >> For example: >> - on CPU node : /tmp/mpich/install/bin holds the CPU version >> - on MIC : /tmp/mpich/install/bin holds the MIC version >> 3. compile helloworld.c with the MIC version mpicc >> 4. execute on CPU node: mpiexe -np 2 -f <hostfile with mic >> hostnames>./helloworld >> >> I think you should be able to follow step 2, but since your helloworld >> binary is also built with different OS, you might want to put it also into >> the same path on two nodes similar as we do for MPICH binaries. >> >> Min >> >> >> On 11/16/16 8:29 AM, Kenneth Raffenetti wrote: >> >>> Have you disabled any and all firewalls on both nodes? It sounds like >>> they are unable to communicate in initialization. >>> >>> Ken >>> >>> On 11/16/2016 07:34 AM, Doha Ehab wrote: >>> >>>> Yes, I built MPICH-3 on both systems and I tried the code on each node >>>> separately and it worked, I tried each node with other nodes that has >>>> the same operating system and it worked as well. >>>> When I try the code on the 2 nodes that have different operating systems >>>> no result or error message appear. >>>> >>>> Regards >>>> Doha >>>> >>>> On Mon, Nov 14, 2016 at 6:25 PM, Kenneth Raffenetti >>>> <raffenet(a)mcs.anl.gov <mailto:[email protected]>> wrote: >>>> >>>> It may be possible to run in such a setup, but it would not be >>>> recommended. Did you build MPICH on both systems you are trying to >>>> run on? What exactly happened when the code didn't work? >>>> >>>> Ken >>>> >>>> >>>> On 11/13/2016 12:36 AM, Doha Ehab wrote: >>>> >>>> Hello, >>>> I tried to run a parallel (Hello World) C code on a cluster >>>> that has 2 >>>> nodes, the nodes have different operating system so the code >>>> did not >>>> work and no results were printed. >>>> How to make such a cluster work? is there is extra steps that >>>> should be >>>> done? >>>> >>>> Regards, >>>> Doha >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> <mailto:[email protected]> >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> <https://lists.mpich.org/mailman/listinfo/discuss> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org <mailto: >>>> discuss(a)mpich.org> >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> <https://lists.mpich.org/mailman/listinfo/discuss> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe:https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Hybrid HPC system
by Min Si 22 Jan '17

22 Jan '17

Hi Doha, Can you please send us the config.log file for each MPICH build and your helloworld source doe ? The config.log file should be under your MPICH build directory where you executed ./configure. Min On 1/21/17 4:53 AM, Doha Ehab wrote: > I have tried what you mentioned in the previous E-mail. > > 1- I have build MPICH for CPU node and ARM node. > 2- Uploaded the binaries on same path on the 2 nodes. > 3- Compiled helloWorld (it sends a number from process zero to all > other processes ) for both nodes. Then tried mpiexec -np 2 -f > <hostfile with mic hostnames>./helloworld > > I got this error > Fatal error in MPI_Recv: Other MPI error, error stack: > MPI_Recv(200)................................: > MPI_Recv(buf=0xbe9460d0, count=1, MPI_INT, src=0, tag=0, > MPI_COMM_WORLD, status=0x1) failed > MPIDI_CH3i_Progress_wait(242)................: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > MPIDI_CH3I_Progress_handle_sock_event(554)...: > MPIDI_CH3_Sockconn_handle_connopen_event(899): unable to find the > process group structure with id <> > > Regards, > Doha > > > On Wed, Nov 16, 2016 at 6:38 PM, Min Si <msi(a)anl.gov > <mailto:[email protected]>> wrote: > > I guess you might need to put all the MPICH binaries (e.g., > hydra_pmi_proxy) to the same path on each node. I have executed > MPICH on Intel MIC chips from the host CPU node where OS are > different. The thing I did was: > 1. build MPICH for both CPU node and MIC on the CPU node (you have > done this step). > 2. upload the MIC binaries to the same path on MIC chip as on the > CPU node > For example: > - on CPU node : /tmp/mpich/install/bin holds the CPU version > - on MIC : /tmp/mpich/install/bin holds the MIC version > 3. compile helloworld.c with the MIC version mpicc > 4. execute on CPU node: mpiexe -np 2 -f <hostfile with mic > hostnames>./helloworld > > I think you should be able to follow step 2, but since your > helloworld binary is also built with different OS, you might want > to put it also into the same path on two nodes similar as we do > for MPICH binaries. > > Min > > > On 11/16/16 8:29 AM, Kenneth Raffenetti wrote: > > Have you disabled any and all firewalls on both nodes? It > sounds like they are unable to communicate in initialization. > > Ken > > On 11/16/2016 07:34 AM, Doha Ehab wrote: > > Yes, I built MPICH-3 on both systems and I tried the code > on each node > separately and it worked, I tried each node with other > nodes that has > the same operating system and it worked as well. > When I try the code on the 2 nodes that have different > operating systems > no result or error message appear. > > Regards > Doha > > On Mon, Nov 14, 2016 at 6:25 PM, Kenneth Raffenetti > <raffenet(a)mcs.anl.gov <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>>> wrote: > > It may be possible to run in such a setup, but it > would not be > recommended. Did you build MPICH on both systems you > are trying to > run on? What exactly happened when the code didn't work? > > Ken > > > On 11/13/2016 12:36 AM, Doha Ehab wrote: > > Hello, > I tried to run a parallel (Hello World) C code on > a cluster > that has 2 > nodes, the nodes have different operating system > so the code did not > work and no results were printed. > How to make such a cluster work? is there is > extra steps that > should be > done? > > Regards, > Doha > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > <https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss>> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > <https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss>> > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Hybrid HPC system
by Doha Ehab 21 Jan '17

21 Jan '17

I have tried what you mentioned in the previous E-mail. 1- I have build MPICH for CPU node and ARM node. 2- Uploaded the binaries on same path on the 2 nodes. 3- Compiled helloWorld (it sends a number from process zero to all other processes ) for both nodes. Then tried mpiexec -np 2 -f <hostfile with mic hostnames>./helloworld I got this error Fatal error in MPI_Recv: Other MPI error, error stack: MPI_Recv(200)................................: MPI_Recv(buf=0xbe9460d0, count=1, MPI_INT, src=0, tag=0, MPI_COMM_WORLD, status=0x1) failed MPIDI_CH3i_Progress_wait(242)................: an error occurred while handling an event returned by MPIDU_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(554)...: MPIDI_CH3_Sockconn_handle_connopen_event(899): unable to find the process group structure with id <> Regards, Doha On Wed, Nov 16, 2016 at 6:38 PM, Min Si <msi(a)anl.gov> wrote: > I guess you might need to put all the MPICH binaries (e.g., > hydra_pmi_proxy) to the same path on each node. I have executed MPICH on > Intel MIC chips from the host CPU node where OS are different. The thing I > did was: > 1. build MPICH for both CPU node and MIC on the CPU node (you have done > this step). > 2. upload the MIC binaries to the same path on MIC chip as on the CPU node > For example: > - on CPU node : /tmp/mpich/install/bin holds the CPU version > - on MIC : /tmp/mpich/install/bin holds the MIC version > 3. compile helloworld.c with the MIC version mpicc > 4. execute on CPU node: mpiexe -np 2 -f <hostfile with mic > hostnames>./helloworld > > I think you should be able to follow step 2, but since your helloworld > binary is also built with different OS, you might want to put it also into > the same path on two nodes similar as we do for MPICH binaries. > > Min > > > On 11/16/16 8:29 AM, Kenneth Raffenetti wrote: > >> Have you disabled any and all firewalls on both nodes? It sounds like >> they are unable to communicate in initialization. >> >> Ken >> >> On 11/16/2016 07:34 AM, Doha Ehab wrote: >> >>> Yes, I built MPICH-3 on both systems and I tried the code on each node >>> separately and it worked, I tried each node with other nodes that has >>> the same operating system and it worked as well. >>> When I try the code on the 2 nodes that have different operating systems >>> no result or error message appear. >>> >>> Regards >>> Doha >>> >>> On Mon, Nov 14, 2016 at 6:25 PM, Kenneth Raffenetti >>> <raffenet(a)mcs.anl.gov <mailto:[email protected]>> wrote: >>> >>> It may be possible to run in such a setup, but it would not be >>> recommended. Did you build MPICH on both systems you are trying to >>> run on? What exactly happened when the code didn't work? >>> >>> Ken >>> >>> >>> On 11/13/2016 12:36 AM, Doha Ehab wrote: >>> >>> Hello, >>> I tried to run a parallel (Hello World) C code on a cluster >>> that has 2 >>> nodes, the nodes have different operating system so the code did >>> not >>> work and no results were printed. >>> How to make such a cluster work? is there is extra steps that >>> should be >>> done? >>> >>> Regards, >>> Doha >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> <mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> <https://lists.mpich.org/mailman/listinfo/discuss> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org <mailto:[email protected] >>> > >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> <https://lists.mpich.org/mailman/listinfo/discuss> >>> >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] FW: Potential MPICH problem
by Richard Warren 20 Jan '17

20 Jan '17

Hi Rob, In my previous email, I mentioned that the failure in MPI_File_set_view that we've observed and reported might be the effect of some prior data-corruption, e.g. a buffer overrun... but I haven't found evidence of that yet. Even more curious, is that the application crash occurs from within an internal Barrier operation in MPI_File_set_view given that an MPI_Barrier call that I added immediately prior to the set_view proceeds without error (though I've used MPI_COMM_WORLD for that "test"). Here's a more detailed trace which shows me stepping thru the MPI_File_set_view/Barrier code: 112 my_rank = MPID_nem_mem_region.rank; (gdb) 119 if (MPID_nem_fbox_is_full((MPID_nem_fbox_common_ptr_t)pbox)) (gdb) OPA_load_acquire_int (ptr=0x7ffff5a91040) at /home/riwarren/Sandbox/mpich-3.2/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h:65 65 /home/riwarren/Sandbox/mpich-3.2/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h: No such file or directory. (gdb) MPID_nem_mpich_send_header (size=48, again=<synthetic pointer>, vc=0xb1b838, buf=0x7fffffffcf10) at ./src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:119 119 if (MPID_nem_fbox_is_full((MPID_nem_fbox_common_ptr_t)pbox)) (gdb) 122 pbox->cell.pkt.mpich.source = MPID_nem_mem_region.local_rank; (gdb) 128 MPIU_Memcpy((void *)pbox->cell.pkt.mpich.p.payload, buf, size); (gdb) 122 pbox->cell.pkt.mpich.source = MPID_nem_mem_region.local_rank; (gdb) 123 pbox->cell.pkt.mpich.datalen = size; (gdb) 124 pbox->cell.pkt.mpich.seqno = vc_ch->send_seqno++; (gdb) 128 MPIU_Memcpy((void *)pbox->cell.pkt.mpich.p.payload, buf, size); (gdb) n 130 OPA_store_release_int(&pbox->flag.value, 1); (gdb) where #0 MPID_nem_mpich_send_header (size=48, again=<synthetic pointer>, vc=0xb1b838, buf=0x7fffffffcf10) at ./src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:130 #1 MPIDI_CH3_iSend (vc=vc@entry=0xb1b838, sreq=0x7ffff769fe88 <MPID_Request_direct+904>, hdr=hdr@entry=0x7fffffffcf10, hdr_sz=48, hdr_sz@entry=32) at src/mpid/ch3/channels/nemesis/src/ch3_isend.c:56 #2 0x00007ffff736e6eb in MPID_Isend (buf=buf@entry=0x0, count=count@entry=0, datatype=datatype@entry=1275068685, rank=rank@entry=1, tag=tag@entry=1, comm=comm@entry=0x7ffff7e27520, context_offset=1, request=0x7fffffffcf98) at src/mpid/ch3/src/mpid_isend.c:115 #3 0x00007ffff73127a4 in MPIC_Sendrecv (sendbuf=sendbuf@entry=0x0, sendcount=sendcount@entry=0, sendtype=sendtype@entry=1275068685, dest=dest@entry=1, sendtag=sendtag@entry=1, recvbuf=recvbuf@entry=0x0, recvcount=0, recvtype=1275068685, source=1, recvtag=1, comm_ptr=0x7ffff7e27520, status=0x7fffffffcfa0, errflag=0x7fffffffd14c) at src/mpi/coll/helper_fns.c:481 #4 0x00007ffff726f18c in MPIR_Barrier_intra (comm_ptr=0x7ffff7e27520, errflag=0x7fffffffd14c) at src/mpi/coll/barrier.c:162 #5 0x00007ffff726f7b2 in MPIR_Barrier (comm_ptr=<optimized out>, errflag=<optimized out>) at src/mpi/coll/barrier.c:291 #6 0x00007ffff726f095 in MPIR_Barrier_impl (comm_ptr=<optimized out>, errflag=errflag@entry=0x7fffffffd14c) at src/mpi/coll/barrier.c:326 #7 0x00007ffff726f26a in barrier_smp_intra (errflag=0x7fffffffd14c, comm_ptr=0x7ffff7e27370) at src/mpi/coll/barrier.c:81 #8 MPIR_Barrier_intra (comm_ptr=0x7ffff7e27370, errflag=0x7fffffffd14c) at src/mpi/coll/barrier.c:146 #9 0x00007ffff726f7b2 in MPIR_Barrier (comm_ptr=<optimized out>, errflag=<optimized out>) at src/mpi/coll/barrier.c:291 #10 0x00007ffff726f095 in MPIR_Barrier_impl (comm_ptr=comm_ptr@entry=0x7ffff7e27370, errflag=errflag@entry=0x7fffffffd14c) at src/mpi/coll/barrier.c:326 #11 0x00007ffff726f9e2 in PMPI_Barrier (comm=-1006632958) at src/mpi/coll/barrier.c:410 #12 0x00007ffff73b6bf1 in PMPI_File_set_view (fh=0xb99e78, disp=0, etype=etype@entry=1275068685, filetype=-1946157049, datarep=<optimized out>, datarep@entry=0xae9b08 <H5FD_mpi_native_g> "native", info=-1677721596) at mpi-io/set_view.c:188 #13 0x000000000075cce1 in H5FD_mpio_write (_file=_file@entry=0xbb21d0, type=type@entry=H5FD_MEM_DEFAULT, dxpl_id=<optimized out>, addr=addr@entry=0, size=size@entry=1, buf=buf@entry=0xbcb830) at H5FDmpio.c:1781 #14 0x000000000055e285 in H5FD_write (file=0xbb21d0, dxpl=0xb84210, type=type@entry=H5FD_MEM_DEFAULT, addr=addr@entry=0, size=size@entry=1, buf=buf@entry=0xbcb830) at H5FDint.c:294 #15 0x000000000054aa12 in H5F__accum_write (fio_info=fio_info@entry=0x7fffffffd350, map_type=map_type@entry=H5FD_MEM_DEFAULT, addr=addr@entry=0, size=size@entry=1, buf=buf@entry=0xbcb830) at H5Faccum.c:821 #16 0x000000000054c5fc in H5F_block_write (f=f@entry=0xbb2290, type=type@entry=H5FD_MEM_DEFAULT, addr=addr@entry=0, size=size@entry=1, dxpl_id=dxpl_id@entry=720575940379279375, buf=buf@entry=0xbcb830) at H5Fio.c:195 #17 0x0000000000752a11 in H5C__collective_write (f=f@entry=0xbb2290, dxpl_id=dxpl_id@entry=720575940379279375) at H5Cmpio.c:1454 #18 0x0000000000754267 in H5C_apply_candidate_list (f=f@entry=0xbb2290, dxpl_id=dxpl_id@entry=720575940379279375, cache_ptr=cache_ptr@entry=0x7ffff455a040, num_candidates=1, candidates_list_ptr=<optimized out>, mpi_rank=<optimized out>, mpi_size=2) at H5Cmpio.c:760 #19 0x0000000000750676 in H5AC__rsp__dist_md_write__flush (f=f@entry=0xbb2290, dxpl_id=dxpl_id@entry=720575940379279375) at H5ACmpio.c:1707 #20 0x0000000000751f7f in H5AC__run_sync_point (f=f@entry=0xbb2290, dxpl_id=dxpl_id@entry=720575940379279375, sync_point_op=sync_point_op@entry=1) at H5ACmpio.c:2158 #21 0x000000000075205e in H5AC__flush_entries (f=f@entry=0xbb2290, dxpl_id=dxpl_id@entry=720575940379279375) at H5ACmpio.c:2301 #22 0x00000000004bfe16 in H5AC_dest (f=f@entry=0xbb2290, dxpl_id=dxpl_id@entry=720575940379279375) at H5AC.c:582 #23 0x0000000000543ef1 in H5F_dest (f=f@entry=0xbb2290, dxpl_id=720575940379279375, flush=flush@entry=true) at H5Fint.c:964 #24 0x0000000000544ae2 in H5F_try_close (f=f@entry=0xbb2290, was_closed=was_closed@entry=0x0) at H5Fint.c:1800 #25 0x0000000000544ee2 in H5F_close (f=0xbb2290) at H5Fint.c:1626 #26 0x00000000005b65cd in H5I_dec_ref (id=id@entry=72057594037927936) at H5I.c:1308 #27 0x00000000005b669e in H5I_dec_app_ref (id=id@entry=72057594037927936) at H5I.c:1353 #28 0x000000000053d058 in H5Fclose (file_id=72057594037927936) at H5F.c:769 #29 0x0000000000487353 in ADFH_Database_Close (root=4.7783097267364807e-299, status=0x7fffffffdb24) at adfh/ADFH.c:2447 #30 0x0000000000481633 in cgio_close_file (cgio_num=1) at cgns_io.c:817 #31 0x00000000004060d7 in cg_close (file_number=1) at cgnslib.c:636 #32 0x0000000000437417 in cgp_close (fn=1) at pcgnslib.c:288 #33 0x0000000000403524 in main (argc=1, argv=0x7fffffffdd28) at benchmark_hdf5.c:186 The code fails when I attempt to step over the OPA_store_release_int function. Might you have some explanation about what things *COULD* go wrong with this? Not knowing the current details of the MPICH collectives, why would a Barrier operation which immediate precedes that call to MPI_File_set_view work correctly and then have that same operation fail from within? Thanks for any insight you have on this issue! Best regards, Richard ________________________________ From: Richard Warren <Richard.Warren(a)hdfgroup.org> Sent: Wednesday, January 11, 2017 11:27:42 AM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] FW: Potential MPICH problem [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing] How Office helps protect you from phishing schemes - Office Support<http://aka.ms/LearnAboutSpoofing> aka.ms This article explains what phishing is and includes tips on how to identify phishing schemes and follow best practices to avoid becoming a victim of online fraud. Hi Rob, I did put together an initial simplified test case to attempt to reproduce the issue, but that does code does NOT fail! I’ve since focused my efforts not so much on the file closing operations, but other collective operations that take place PRIOR to the close. I suspect that some operation(s) that precede the file close are responsible for potentially getting things out-of-sync and hence we observe the failure “down-stream”. At this time, I’m spending my time attempting to trace these other collective operations in order to gain a better understanding what’s happening. The code that we’re testing has not been released yet, so I’m not convinced that you could reproduce it directly from the currently available HDF5 and CGNS downloads. I could send you the actual test, since it links to the static libraries (.a) for both HDF5 and CGNS. Would that be of interest to you? Thanks, Richard On 1/11/17, 11:02 AM, "Rob Latham" <robl(a)mcs.anl.gov> wrote: On 12/29/2016 08:28 AM, Richard Warren wrote: > > > Hi All, > > I’m writing to get some advice and possibly report a bug. The > circumstances are that we are currently working on updating HDF5 > functionality and have run into an issue running a parallel test of a > CFD code (benchmark.hdf) from the CGNS code base > <https://github.com/CGNS/CGNS.git>. I’ve debugged enough to see that CGNS/CGNS<https://github.com/CGNS/CGNS.git> github.com The CFD General Notation System (CGNS) provides a standard for recording and recovering computer data associated with the numerical solution of fluid dynamics equations. All developement work and b... > our failure occurs during a call to MPI_File_set_view, with the failure > signature as follows: i'm having a heck of a time building CGNS. When I make cgns, cgnsconvert is failing to find the hdf5 symbols despite me telling cmake where to find libhdf5. did you get that test case you mentioned? ==rob > > > > [brtnfld@jelly] ~/scratch/CGNS/CGNS/src/ptests % mpirun -n 2 benchmark_hdf5 > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > MPIR_Barrier_impl(337)..............: Failure during collective > > MPIR_Barrier_impl(330)..............: > > MPIR_Barrier(294)...................: > > MPIR_Barrier_intra(151).............: > > barrier_smp_intra(111)..............: > > MPIR_Bcast_impl(1462)...............: > > MPIR_Bcast(1486)....................: > > MPIR_Bcast_intra(1295)..............: > > MPIR_Bcast_binomial(241)............: > > MPIC_Recv(352)......................: > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > received but buffer size is 1 > > [cli_1]: aborting job: > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > MPIR_Barrier_impl(337)..............: Failure during collective > > MPIR_Barrier_impl(330)..............: > > MPIR_Barrier(294)...................: > > MPIR_Barrier_intra(151).............: > > barrier_smp_intra(111)..............: > > MPIR_Bcast_impl(1462)...............: > > MPIR_Bcast(1486)....................: > > MPIR_Bcast_intra(1295)..............: > > MPIR_Bcast_binomial(241)............: > > MPIC_Recv(352)......................: > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > received but buffer size is 1 > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > H5F_close: Assertion `f->file_id > 0' failed. > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > PMPI_Allgather(1002)......................: > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > MPIR_Allgather_impl(842)..................: > > MPIR_Allgather(801).......................: > > MPIR_Allgather_intra(216).................: > > MPIC_Sendrecv(475)........................: > > MPIC_Wait(243)............................: > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > > MPIDI_CH3I_Progress_handle_sock_event(451): > > MPIDU_Socki_handle_read(649)..............: connection failure > (set=0,sock=1,errno=104:Connection reset by peer) > > [cli_0]: aborting job: > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > PMPI_Allgather(1002)......................: > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > MPIR_Allgather_impl(842)..................: > > MPIR_Allgather(801).......................: > > MPIR_Allgather_intra(216).................: > > MPIC_Sendrecv(475)........................: > > MPIC_Wait(243)............................: > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > > MPIDI_CH3I_Progress_handle_sock_event(451): > > MPIDU_Socki_handle_read(649)..............: connection failure > (set=0,sock=1,errno=104:Connection reset by peer) > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > H5F_close: Assertion `f->file_id > 0’ failed. > > > > Please note that the above trace was the original stacktrace which > appears to utilize sockets, though I’ve reproduced the same problem by > running on an SMP with shared memory. > > > > While it’s not definitive that the issue has anything to do with the > above stack trace, the very same benchmark runs perfectly well utilizing > PHDF5 built with OpenMPI. My own testing is with MPICH version 3.2 > available from your download site and with OpenMPI 2.0.1 (also their > latest download). Both MPI releases were built from source on my Fedora > 25 Linux distribution using GCC 6.2.1 20160916 (Red Hat 6.2.1-2). > > > > Given that the synchronous calls into MPI_File_set_view appear to be > coded correctly AND that there isn’t much in the way of input parameters > that would cause problems (other than incorrect coding), we tend to > believe that the internal message queues between processes may somehow > be corrupted. This impression is strengthened by the fact that our > recent codebase changes (which are unrelated to the actual calls to > MPI_File_set_view) may have introduced this issue. Note too, that the > code paths to MPI_File_set_view have been taken many times previously > and those function calls have all succeeded. > > > > Are there any suggestions out there as to how to further debug this > potential corruption issue? > > Many thanks, > > Richard A. Warren > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss discuss Info Page - MPICH<https://lists.mpich.org/mailman/listinfo/discuss> lists.mpich.org To see the collection of prior postings to the list, visit the discuss Archives. Using discuss: To post a message to all the list members, send email ... > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss discuss Info Page - MPICH<https://lists.mpich.org/mailman/listinfo/discuss> lists.mpich.org To see the collection of prior postings to the list, visit the discuss Archives. Using discuss: To post a message to all the list members, send email ... _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss discuss Info Page - MPICH<https://lists.mpich.org/mailman/listinfo/discuss> lists.mpich.org To see the collection of prior postings to the list, visit the discuss Archives. Using discuss: To post a message to all the list members, send email ... _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] spurious lock ups on collective merge intercom
by Dmitriy Lyubimov 19 Jan '17

19 Jan '17

These lock-ups seem to be gone in 3.3a2. I do occasionally get the following though: Unknown error class, error stack: PMPI_Comm_accept(129).................: MPI_Comm_accept(port="tag#0$description#aaa.com$port#36230$ifname#192.168.42.99$", MPI_INFO_NULL, ro ot=180, comm=0x84000003, newcomm=0x7f3cf681842c) failed MPID_Comm_accept(153).................: MPIDI_Comm_accept(1244)...............: MPIR_Get_contextid_sparse_group(499)..: MPIR_Allreduce_impl(755)..............: MPIR_Allreduce_intra(414).............: MPIDU_Complete_posted_with_error(1137): Process failed What does this message mean? some process just exited/died (like with seg fault?) Thank you. -Dmitriy On Thu, Jan 12, 2017 at 11:55 AM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> wrote: > further debugging shows that it's not actually mergeIntercom that locks up > but a pair of send/recv that two nodes decide to execute before > MPI_intercom_merge. > > so the total snapshot of the situation is that everyone waits on > mergeIntercom except for two processes that wait in send/recv respectively, > while majority of others already have entered collective barrier. > > it would seem that this sort of assymetric logic would be acceptable, > since the send/recv pair is balanced before the merge is to occur, but in > practice it seems to lock up -- increasingly so as the number of > participating processes increases. It almost like once collective barrier > of certain cardinality is formed, point-to-point messages are not going > thru any longer. > > If this scenario begets any ideas, please let me know. > > thank you! > -Dmitriy > > > > On Wed, Jan 11, 2017 at 9:38 AM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> > wrote: > >> Maybe it has something to do with the fact that it is stepping thru JVM >> JNI and that somehow screws threading model of MPI, although it is a single >> threaded JVM process, and MPI mappings are known to have been done before >> (e.g., openmpi had an effort towards that). >> >> Strange thing is that i never had lock up with # of processes under 120 >> but something changes after that, the spurious condition becomes much more >> common after that. By the time I am at 150 processes in the intercom, I am >> almost certain to have a merge lock-up. >> >> >> On Wed, Jan 11, 2017 at 9:34 AM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> >> wrote: >> >>> Thanks. >>> it would not be easy for me to do immediately as i am using proprietary >>> scala binding api for MPI. >>> >>> it would help me to know if there's a known problem like that in the >>> past, or generally mergeIntercomm api is known to work on hundreds of >>> processes. Sounds like there are no known issues with that. >>> >>> >>> >>> On Tue, Jan 10, 2017 at 11:53 PM, Oden, Lena <loden(a)anl.gov> wrote: >>> >>>> Hello Dmittiy, >>>> >>>> can you maybe create a simple example-program to reproduce this failure? >>>> It is also often easier also to look at a code example to identify a >>>> problem. >>>> >>>> Thanks, >>>> Lena >>>> > On Jan 11, 2017, at 2:45 AM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> >>>> wrote: >>>> > >>>> > Hello, >>>> > >>>> > (mpich 3.2) >>>> > >>>> > I have a scenario when i add a few extra processes do existing >>>> intercom. >>>> > >>>> > it works as a simple loop -- >>>> > (1) n processes accept on n-intercom >>>> > (2) 1 process connects >>>> > (3) intracom is merged into n+1 intercom, intracom and n-intercom are >>>> closed >>>> > (4) repeat 1-3 as needed. >>>> > >>>> > Occasionally, i observe that step 3 spuriously locks up (once i get >>>> in the range of 100+ processes). From what i can tell, all processes in >>>> step 3 are accounted for, and are waiting on the merge, but nothing >>>> happens. the collective barrier locks up. >>>> > >>>> > I really have trouble resolving this issue, any ideas are appreciated! >>>> > >>>> > Thank you very much. >>>> > -Dmitriy >>>> > >>>> > >>>> > _______________________________________________ >>>> > discuss mailing list discuss(a)mpich.org >>>> > To manage subscription options or unsubscribe: >>>> > https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >>> >> > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 and Nagfor: Corrupts writing/IO?
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 18 Jan '17

18 Jan '17

Ken, et al, I have found one more possible issue with NAG. It seems to be due to the use of, in MPICH, statements like: integer*8 real*8 Why? Well, by default, NAG really wants those to be integer*2 and real*2 because their default is -kind=sequential (as kind numbers are implementation dependent). I'm currently trying to build another version of MPICH3 by passing in -kind=byte (4,8,... instead of 1,2...) because I can't build any upstream libraries (e.g., HDF5) with -kind=byte if the MPI wasn't built that way. Fun! Now, it might be impossible to change such MPI modules to use 'selected_real_kind' and 'selected_int_kind' (as is allowed in F90), but, it might be good if anyone cared to. The one I'm more concerned with is seeing "integer*8" in: test/mpi/f08/coll/allredopttf08.f90 That is part of the F08 interface. Once you are F08, you could (should?) just use INT64, REAL32, etc. like is done in: src/binding/fortran/use_mpi_f08/mpi_f08_types.f90 or selected_real/int_kind by default. Still, no matter what, I'm happy I have a working stack! Matt On 01/17/2017 01:18 PM, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] wrote: > Ken, > > I just tried your configure (with enable-cxx) and it seemed to work! > Thanks! > > I suppose I should try and figure out what flag caused the corruption > but it's working. Don't really want to poke the bear right now. > > Matt > > > On 01/12/2017 04:46 PM, Kenneth Raffenetti wrote: >> Can you check your compiler flags and make sure they are all necessary? >> I was able to reproduce the error locally with your settings, but a more >> default configuration works fine. I.e. >> >> ./configure --prefix=$PWD/i --disable-wrapper-rpath CC=gcc CXX=g++ >> FC=nagfor F77=nagfor FCFLAGS=-mismatch FFLAGS=-mismatch >> --enable-fortran=all >> >> Ken >> >> On 01/12/2017 10:54 AM, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND >> APPLICATIONS INC] wrote: >>> All, >>> >>> I've been having some "fun" recently trying to get an MPI stack built >>> with nagfor 6.1. I've tried Open MPI and MVAPICH2 and failed to even >>> build the MPI stack, SGI MPT doesn't like nagfor, and Intel MPI I'm >>> guessing wouldn't either. >>> >>> So, I figured I'd go for MPICH3. And, lo and behold, building with >>> nagfor 6.1 and gcc 5.3 (for CC and CXX) with: >>> >>>> ./configure >>>> --prefix=$SWDEV/MPI/mpich/3.2/nagfor_6.1-gcc_5.3-nomismatchall \ >>> --disable-wrapper-rpath CC=gcc CXX=g++ FC=nagfor F77=nagfor \ >>> CFLAGS='-fpic -m64' CXXFLAGS='-fpic -m64' \ >>> FCFLAGS='-PIC -abi=64' FFLAGS='-PIC -abi=64 -mismatch' \ >>> --enable-fortran=all --enable-cxx >>> >>> I got something to build. Huzzah! >>> >>> I then tried the cpi test, it worked! It even detected I was on slurm >>> according to mpirun -verbose. >>> >>> I then tried a simple Fortran 90 Hello world program and...crash: >>> >>>> (1211)(master) $ cat helloWorld.F90 >>>> program hello_world >>>> >>>> use mpi >>>> >>>> implicit none >>>> >>>> integer :: comm >>>> integer :: myid, npes, ierror >>>> integer :: name_length >>>> >>>> character(len=MPI_MAX_PROCESSOR_NAME) :: processor_name >>>> >>>> call mpi_init(ierror) >>>> >>>> comm = MPI_COMM_WORLD >>>> >>>> call MPI_Comm_Rank(comm,myid,ierror) >>>> call MPI_Comm_Size(comm,npes,ierror) >>>> call MPI_Get_Processor_Name(processor_name,name_length,ierror) >>>> >>>> write (*,'(A,1X,I4,1X,A,1X,I4,1X,A,1X,A)') "Process", myid, "of", >>>> npes, "is on", trim(processor_name) >>>> >>>> call MPI_Finalize(ierror) >>>> >>>> end program hello_world >>>> (1212)(master) $ mpifort -o helloWorld.exe helloWorld.F90 >>>> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >>>> [NAG Fortran Compiler normal termination] >>>> (1213)(master) $ mpirun -np 4 ./helloWorld.exe >>>> srun.slurm: cluster configuration lacks support for cpu binding >>>> Runtime Error: Buffer overflow on output >>>> Program terminated by I/O error on unit 6 >>>> (Output_Unit,Unformatted,Direct) >>>> Runtime Error: Buffer overflow on output >>>> Program terminated by I/O error on unit 6 >>>> (Output_Unit,Unformatted,Direct) >>>> Runtime Error: Buffer overflow on output >>>> Program terminated by I/O error on unit Runtime Error: Buffer overflow >>>> on output >>>> Program terminated by I/O error on unit 6 >>>> (Output_Unit,Unformatted,Direct) >>>> 6 (Output_Unit,Unformatted,Direct) >>>> >>>> =================================================================================== >>>> >>>> >>>> >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = PID 7642 RUNNING AT borgl189 >>>> = EXIT CODE: 134 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> =================================================================================== >>>> >>>> >>>> >>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) >>>> This typically refers to a problem with your application. >>>> Please see the FAQ page for debugging suggestions >>> >>> Weird. So, I decided to try something different, this program: >>> >>>> program main >>>> implicit none >>>> real :: a >>>> a = 1240.0 >>>> write (*,*) "Hello world", a >>>> end program main >>> >>> Looks boring and is standard-compliant and nagfor likes it: >>> >>>> (1226) $ nagfor test.F90 && ./a.out >>>> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >>>> [NAG Fortran Compiler normal termination] >>>> Hello world 1.2400000E+03 >>> >>> Looks correct. Now let's try mpifort: >>> >>>> (1232) $ mpifort test.F90 && ./a.out >>>> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >>>> [NAG Fortran Compiler normal termination] >>>> Hello world >>>> Segmentation fault (core dumped) >>> >>> You can't really see it here, but that "Hello world" is surrounded by LF >>> characters. Like a literal LineFeed...and then it core dumps. >>> >>> Now let's try running with mpirun as well: >>> >>>> (1233) $ mpifort test.F90 && mpirun -np 1 ./a.out >>>> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >>>> [NAG Fortran Compiler normal termination] >>>> srun.slurm: cluster configuration lacks support for cpu binding >>>> >>>> =================================================================================== >>>> >>>> >>>> >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = PID 8520 RUNNING AT borgl189 >>>> = EXIT CODE: 139 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> =================================================================================== >>>> >>>> >>>> >>>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault >>>> (signal 11) >>>> This typically refers to a problem with your application. >>>> Please see the FAQ page for debugging suggestions >>> >>> All righty then. >>> >>> Does anyone have advice for this? I'll fully accept I configured MPICH3 >>> wrong as it's the first time in a while I've built MPICH (think MPICH2). >>> But, still, I don't have any exciting flags. >>> >>> Matt >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0