June 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] mpi_f_sync_reg
by Jeff Hammond 25 Jun '16

25 Jun '16

https://trac.mpich.org/projects/mpich/ticket/2342 was created for this. Jeff On Tue, Jun 21, 2016 at 2:47 AM, C.Friedrich <c.friedrich(a)fz-juelich.de> wrote: > Dear all, > > in MPICH 3.2 (using "# include 'mpif.h' and "use mpi") I am missing the > routine > mpi_f_sync_reg > which avoids problems with the optimization of Fortran compilers (e.g., > assignment movements across MPI_WIN_FENCE). It seems to be available in > the new module "mpi_f08", though. Well, I guess sooner or later I will > have to switch to this Fortran 2008 compliant module... > > Regards > Christoph > > > > > ------------------------------------------------------------------------------------------------ > > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > > ------------------------------------------------------------------------------------------------ > > ------------------------------------------------------------------------------------------------ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Mpich-3.2 slowdown with gforker
by Jeff Hammond 22 Jun '16

22 Jun '16

Why don't you use hydra instead of gforker? Over subscribing is bad if MPI polls in shared memory code. I believe nemesis does this. You can use ch3:sock to workaround but it is discouraged. Best option is to not oversubscribe. Do you need this for more than debugging? I know PETSc folks like to debug that way. Is hyper threading turned on or off? /proc/cpuinfo will answer this if you don't know. It is on by default AFAIK. Jeff On Wednesday, June 22, 2016, Shemon, Emily R. <eshemon(a)anl.gov> wrote: > Hello, > > > > We used to use mpich2-1.2 with gforker process manager (Intel 10.1 > compiler). > > We are transitioning mpich-3.2 with gforker process manager (Intel 13.1 > compiler) (configuration options are the same as before: ./configure > --prefix=$MPI_DIR --enable-g=debug,meminit --enable-fast=O2 > --with-pm=gforker --enable-cxx --enable-fortran CC=icc CXX=icpc FC=ifort > F77=ifort) > > We have noticed extreme slowdowns with the new installation of mpich on > the same machine (shared memory Linux server with CentOS 7) when attempting > to use hyperthreading. The old executable works just fine on the same > machine, but the new executable using mpich-3.2 is running 10-20x slower > when going past the number of physical cores on the machine. We believe we > have traced it to our MPI_AllReduce functions. This is a Fortran code. > > > > In the meantime I’m going to try to install the old mpich2-1.2 again (we > had a machine OS upgrade) and see what happens. > > > > Any ideas? > > > > Thank you, > > > > Emily Shemon, Ph.D. > > Argonne National Laboratory > > Nuclear Engineering Division & > > Argonne Leadership Computing Facility > > P: 630-252-4477 > > E: eshemon(a)anl.gov <javascript:_e(%7B%7D,'cvml','eshemon(a)anl.gov');> > > > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Mpich-3.2 slowdown with gforker
by Shemon, Emily R. 22 Jun '16

22 Jun '16

Hello, We used to use mpich2-1.2 with gforker process manager (Intel 10.1 compiler). We are transitioning mpich-3.2 with gforker process manager (Intel 13.1 compiler) (configuration options are the same as before: ./configure --prefix=$MPI_DIR --enable-g=debug,meminit --enable-fast=O2 --with-pm=gforker --enable-cxx --enable-fortran CC=icc CXX=icpc FC=ifort F77=ifort) We have noticed extreme slowdowns with the new installation of mpich on the same machine (shared memory Linux server with CentOS 7) when attempting to use hyperthreading. The old executable works just fine on the same machine, but the new executable using mpich-3.2 is running 10-20x slower when going past the number of physical cores on the machine. We believe we have traced it to our MPI_AllReduce functions. This is a Fortran code. In the meantime I'm going to try to install the old mpich2-1.2 again (we had a machine OS upgrade) and see what happens. Any ideas? Thank you, Emily Shemon, Ph.D. Argonne National Laboratory Nuclear Engineering Division & Argonne Leadership Computing Facility P: 630-252-4477 E: eshemon(a)anl.gov<mailto:[email protected]> _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem with Mpich3.0.4 build for WRF run across multiple nodes in a cluster.
by Rob Latham 22 Jun '16

22 Jun '16

On 06/21/2016 10:09 PM, Teck-Bin Arthur Lim wrote: > Dear Rob, > > Thank you for your pointer. It was down to simple machinefile syntax error, > mpich is fine. I got it working across multiple nodes. that's right. the 'cpu=3' notation does not work. Where did you get the idea to do that? If it's in some documentation we have, we'd like to correct it. For anyone coming here via google.. the correct way to tell mpich that you have, say, 6 cpus in a node is with a colon. Here's an example: $ cat machinefile node1:6 node2:6 node3:1 https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager ==rob > > Arthur > > -----Original Message----- > From: Teck-Bin Arthur Lim [mailto:[email protected]] > Sent: Wednesday, June 22, 2016 10:23 AM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Problem with Mpich3.0.4 build for WRF run across multiple nodes in a cluster. > > Dear Rob, > > Thanks, my machinefile and script are simple, as follows: > ********************************************* > [limtba@fdns00_ws testenv-testlib]$ more mpi-script nohup time mpirun -machinefile machinefile -np 8 ./a.out >& log.aout & > > [limtba@fdns00_ws testenv-testlib]$ more machinefile fdns00_ib cpu=3 fdns01_ib cpu=2 fdns02_ib cpu=3 > ********************************************* > Is there some syntax error? This is only a small test. > Without machinefile, mpich works fine on sinlge node. > > Arthur > > -----Original Message----- > From: Rob Latham [mailto:[email protected]] > Sent: Tuesday, June 21, 2016 11:21 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Problem with Mpich3.0.4 build for WRF run across multiple nodes in a cluster. > > > > On 06/21/2016 05:58 AM, Teck-Bin Arthur Lim wrote: >> Hi , >> >> I met with a basic problem trying to get old mpich(3.0.4) version to do >> parallel run across different machines in a mini-cluster. The error >> messages >> >> while invoking the mpiruns are : >> >> *****************error messages******************** >> >> [limtba@fdns00_ws testenv-testlib]$ ./mpi-script >> >> [limtba@fdns00_ws testenv-testlib]$ more log.aout >> >> [mpiexec@fdns00_ws] HYDU_process_mfile_token (./utils/args/args.c:299): >> token cpu not supported at this time >> >> [mpiexec@fdns00_ws] HYDU_parse_hostfile (./utils/args/args.c:347): >> unable to process token > > what does your machine file look like? it sounds like you've got something in there that Hydra does not expect. > > ==rob >> >> [mpiexec@fdns00_ws] mfile_fn (./ui/mpich/utils.c:341): error parsing >> hostfile >> >> [mpiexec@fdns00_ws] match_arg (./utils/args/args.c:153): match handler >> returned error >> >> [mpiexec@fdns00_ws] HYDU_parse_array (./utils/args/args.c:175): >> argument matching returned error >> >> [mpiexec@fdns00_ws] parse_args (./ui/mpich/utils.c:1609): error >> parsing input array >> >> [mpiexec@fdns00_ws] HYD_uii_mpx_get_parameters >> (./ui/mpich/utils.c:1660): unable to parse user arguments >> >> [mpiexec@fdns00_ws] main (./ui/mpich/mpiexec.c:153): error parsing >> parameters >> >> Command exited with non-zero status 255 >> >> *****************error messages******************** >> >> This old version was downloaded from WRF site >> (http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php# >> STEP2) >> , and was built >> >> with essentially, all the default configuration settings without any >> options arguments given in the configure/make/make-install process, as : >> >> Ø./configure -prefix=$DIR/mpich >> >> Ømake >> >> Ømake install >> >> There are no error messages during the built process, and the mpirun >> works fine for parallel runs, using multiple processors, on a single >> NODE only >> >> but met with the above error messages when attempting parallel run >> across multiple machines. >> >> I need some advice as to how get this old mpich 3.0.4 working across >> machines. The OS for these machines are Centos5.5, with gcc4.1.2 and >> >> gcc4.4.7 installations. As WRF needs gcc4.4 and higher version, I >> have built the mpich3.04 using gcc4.4.7. >> >> Would appreciate any help and advices. >> >> Many Thanks. >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem with Mpich3.0.4 build for WRF run across multiple nodes in a cluster.
by Teck-Bin Arthur Lim 21 Jun '16

21 Jun '16

Dear Rob, Thank you for your pointer. It was down to simple machinefile syntax error, mpich is fine. I got it working across multiple nodes. Arthur -----Original Message----- From: Teck-Bin Arthur Lim [mailto:[email protected]] Sent: Wednesday, June 22, 2016 10:23 AM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Problem with Mpich3.0.4 build for WRF run across multiple nodes in a cluster. Dear Rob, Thanks, my machinefile and script are simple, as follows: ********************************************* [limtba@fdns00_ws testenv-testlib]$ more mpi-script nohup time mpirun -machinefile machinefile -np 8 ./a.out >& log.aout & [limtba@fdns00_ws testenv-testlib]$ more machinefile fdns00_ib cpu=3 fdns01_ib cpu=2 fdns02_ib cpu=3 ********************************************* Is there some syntax error? This is only a small test. Without machinefile, mpich works fine on sinlge node. Arthur -----Original Message----- From: Rob Latham [mailto:[email protected]] Sent: Tuesday, June 21, 2016 11:21 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Problem with Mpich3.0.4 build for WRF run across multiple nodes in a cluster. On 06/21/2016 05:58 AM, Teck-Bin Arthur Lim wrote: > Hi , > > I met with a basic problem trying to get old mpich(3.0.4) version to do > parallel run across different machines in a mini-cluster. The error > messages > > while invoking the mpiruns are : > > *****************error messages******************** > > [limtba@fdns00_ws testenv-testlib]$ ./mpi-script > > [limtba@fdns00_ws testenv-testlib]$ more log.aout > > [mpiexec@fdns00_ws] HYDU_process_mfile_token (./utils/args/args.c:299): > token cpu not supported at this time > > [mpiexec@fdns00_ws] HYDU_parse_hostfile (./utils/args/args.c:347): > unable to process token what does your machine file look like? it sounds like you've got something in there that Hydra does not expect. ==rob > > [mpiexec@fdns00_ws] mfile_fn (./ui/mpich/utils.c:341): error parsing > hostfile > > [mpiexec@fdns00_ws] match_arg (./utils/args/args.c:153): match handler > returned error > > [mpiexec@fdns00_ws] HYDU_parse_array (./utils/args/args.c:175): > argument matching returned error > > [mpiexec@fdns00_ws] parse_args (./ui/mpich/utils.c:1609): error > parsing input array > > [mpiexec@fdns00_ws] HYD_uii_mpx_get_parameters > (./ui/mpich/utils.c:1660): unable to parse user arguments > > [mpiexec@fdns00_ws] main (./ui/mpich/mpiexec.c:153): error parsing > parameters > > Command exited with non-zero status 255 > > *****************error messages******************** > > This old version was downloaded from WRF site > (http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php# > STEP2) > , and was built > > with essentially, all the default configuration settings without any > options arguments given in the configure/make/make-install process, as : > > Ø./configure -prefix=$DIR/mpich > > Ømake > > Ømake install > > There are no error messages during the built process, and the mpirun > works fine for parallel runs, using multiple processors, on a single > NODE only > > but met with the above error messages when attempting parallel run > across multiple machines. > > I need some advice as to how get this old mpich 3.0.4 working across > machines. The OS for these machines are Centos5.5, with gcc4.1.2 and > > gcc4.4.7 installations. As WRF needs gcc4.4 and higher version, I > have built the mpich3.04 using gcc4.4.7. > > Would appreciate any help and advices. > > Many Thanks. > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem with Mpich3.0.4 build for WRF run across multiple nodes in a cluster.
by Teck-Bin Arthur Lim 21 Jun '16

21 Jun '16

Dear Rob, Thanks, my machinefile and script are simple, as follows: ********************************************* [limtba@fdns00_ws testenv-testlib]$ more mpi-script nohup time mpirun -machinefile machinefile -np 8 ./a.out >& log.aout & [limtba@fdns00_ws testenv-testlib]$ more machinefile fdns00_ib cpu=3 fdns01_ib cpu=2 fdns02_ib cpu=3 ********************************************* Is there some syntax error? This is only a small test. Without machinefile, mpich works fine on sinlge node. Arthur -----Original Message----- From: Rob Latham [mailto:[email protected]] Sent: Tuesday, June 21, 2016 11:21 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Problem with Mpich3.0.4 build for WRF run across multiple nodes in a cluster. On 06/21/2016 05:58 AM, Teck-Bin Arthur Lim wrote: > Hi , > > I met with a basic problem trying to get old mpich(3.0.4) version to do > parallel run across different machines in a mini-cluster. The error > messages > > while invoking the mpiruns are : > > *****************error messages******************** > > [limtba@fdns00_ws testenv-testlib]$ ./mpi-script > > [limtba@fdns00_ws testenv-testlib]$ more log.aout > > [mpiexec@fdns00_ws] HYDU_process_mfile_token (./utils/args/args.c:299): > token cpu not supported at this time > > [mpiexec@fdns00_ws] HYDU_parse_hostfile (./utils/args/args.c:347): > unable to process token what does your machine file look like? it sounds like you've got something in there that Hydra does not expect. ==rob > > [mpiexec@fdns00_ws] mfile_fn (./ui/mpich/utils.c:341): error parsing > hostfile > > [mpiexec@fdns00_ws] match_arg (./utils/args/args.c:153): match handler > returned error > > [mpiexec@fdns00_ws] HYDU_parse_array (./utils/args/args.c:175): > argument matching returned error > > [mpiexec@fdns00_ws] parse_args (./ui/mpich/utils.c:1609): error > parsing input array > > [mpiexec@fdns00_ws] HYD_uii_mpx_get_parameters > (./ui/mpich/utils.c:1660): unable to parse user arguments > > [mpiexec@fdns00_ws] main (./ui/mpich/mpiexec.c:153): error parsing > parameters > > Command exited with non-zero status 255 > > *****************error messages******************** > > This old version was downloaded from WRF site > (http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php# > STEP2) > , and was built > > with essentially, all the default configuration settings without any > options arguments given in the configure/make/make-install process, as : > > Ø./configure -prefix=$DIR/mpich > > Ømake > > Ømake install > > There are no error messages during the built process, and the mpirun > works fine for parallel runs, using multiple processors, on a single > NODE only > > but met with the above error messages when attempting parallel run > across multiple machines. > > I need some advice as to how get this old mpich 3.0.4 working across > machines. The OS for these machines are Centos5.5, with gcc4.1.2 and > > gcc4.4.7 installations. As WRF needs gcc4.4 and higher version, I > have built the mpich3.04 using gcc4.4.7. > > Would appreciate any help and advices. > > Many Thanks. > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Single node IO Aggregator setup
by Rob Latham 21 Jun '16

21 Jun '16

On 06/21/2016 12:48 PM, Jon Povich wrote: > I'm trying to simulate a cluster setup where only the I/O aggregators > have access to the working directory. Is this feasible to do with romio > hints? > > Setup: > A) Work dir is on Node0's local hard drive. > B) Remote Node1 has no access to Node0's hard drive > C) Run a case where only rank 0 on Node0 serves as the I/O aggregator that should work as long as you only use collective operations. You'll trigger the "deferred open" optimization. https://press3.mcs.anl.gov/romio/2003/08/05/deferred-open/ Your hints are close. > MPI_Info_create(&info); > MPI_Info_set(info, "cb_nodes", "1"); > MPI_Info_set(info, "no_indep_rw", "true"); The hint is actually "romio_no_indep_rw". Where did you find "no_indep_rw" documented? I'd like to get that fixed. I'd adjust one more hint: the "cb_config_list" hint. By default it is "*:1", which means select one MPI process from each node to act as aggregator. That's *probably* node0/crane/whatever, but you can ensure the desired node is selected by being more explicit. I'd make your cb_config_list look like "crane:1" or "node0:1" or whatever MPI_Get_processor_name returns. In 2014 we went on a deferred open cleanup and fixed a few bugs. I hope you are using MPICH-3.1.3 or newer. ==rob _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Single node IO Aggregator setup
by Thakur, Rajeev 21 Jun '16

21 Jun '16

Try running the cpi example from the examples directory across nodes and see if that works. There may be some issue unrelated to MPI-IO. Rajeev > On Jun 21, 2016, at 12:48 PM, Jon Povich <jon.povich(a)convergecfd.com> wrote: > > I'm trying to simulate a cluster setup where only the I/O aggregators have access to the working directory. Is this feasible to do with romio hints? > > Setup: > A) Work dir is on Node0's local hard drive. > B) Remote Node1 has no access to Node0's hard drive > C) Run a case where only rank 0 on Node0 serves as the I/O aggregator > > Simple MPI I/O Code: > > #include "mpi.h" > > #include <string.h> > #include <stdio.h> > #include <unistd.h> > > int main(int argc, char *argv[]) > { > MPI_File fh; > MPI_Info info; > int amode, mpi_ret; > char filename[64]; > > MPI_Init(&argc, &argv); > > > MPI_Info_create(&info); > MPI_Info_set(info, "cb_nodes", "1"); > MPI_Info_set(info, "no_indep_rw", "true"); > > strcpy(filename, "mpio_test.out"); > amode = MPI_MODE_CREATE | MPI_MODE_WRONLY; > mpi_ret = MPI_File_open(MPI_COMM_WORLD, filename, amode, info, &fh); > > if(mpi_ret != MPI_SUCCESS) > { > char mpi_err_buf[MPI_MAX_ERROR_STRING]; > int mpi_err_len; > > MPI_Error_string(mpi_ret, mpi_err_buf, &mpi_err_len); > fprintf(stderr, "Failed MPI_File_open. Filename = %s, error = %s", filename, mpi_err_buf); > return -1; > } > > // Force I/O errors associated with this file to abort > MPI_File_set_errhandler(fh, MPI_ERRORS_ARE_FATAL); > > MPI_File_close(&fh); > > printf("SUCCESS\n"); > > MPI_Finalize(); > > return 0; > } > > Note the hardcoded "MPI_Info_set(info, "cb_nodes", "1");" and "MPI_Info_set(info, "no_indep_rw", "true");". > > The above code runs fine when run from multiple cores on Node0. As soon as I add a Node1 to the mix, I get the following error: > > jpovich@crane mini_test> mpirun -np 2 -hosts crane,node1 /work/jpovich/box_testing/1970/1970_test > [mpiexec(a)crane.csi.com] control_cb (pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed > [mpiexec(a)crane.csi.com] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status > [mpiexec(a)crane.csi.com] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > [mpiexec(a)crane.csi.com] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion > > The cb_nodes setting seems to have no impact on behavior. I get the same error if I comment out the cb_nodes and no_indep_rw settings. > > Any help is greatly appreciated, > > Jon > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Single node IO Aggregator setup
by Jon Povich 21 Jun '16

21 Jun '16

I'm trying to simulate a cluster setup where only the I/O aggregators have access to the working directory. Is this feasible to do with romio hints? Setup: A) Work dir is on Node0's local hard drive. B) Remote Node1 has no access to Node0's hard drive C) Run a case where only rank 0 on Node0 serves as the I/O aggregator Simple MPI I/O Code: #include "mpi.h" #include <string.h> #include <stdio.h> #include <unistd.h> int main(int argc, char *argv[]) { MPI_File fh; MPI_Info info; int amode, mpi_ret; char filename[64]; MPI_Init(&argc, &argv); MPI_Info_create(&info); MPI_Info_set(info, "cb_nodes", "1"); MPI_Info_set(info, "no_indep_rw", "true"); strcpy(filename, "mpio_test.out"); amode = MPI_MODE_CREATE | MPI_MODE_WRONLY; mpi_ret = MPI_File_open(MPI_COMM_WORLD, filename, amode, info, &fh); if(mpi_ret != MPI_SUCCESS) { char mpi_err_buf[MPI_MAX_ERROR_STRING]; int mpi_err_len; MPI_Error_string(mpi_ret, mpi_err_buf, &mpi_err_len); fprintf(stderr, "Failed MPI_File_open. Filename = %s, error = %s", filename, mpi_err_buf); return -1; } // Force I/O errors associated with this file to abort MPI_File_set_errhandler(fh, MPI_ERRORS_ARE_FATAL); MPI_File_close(&fh); printf("SUCCESS\n"); MPI_Finalize(); return 0; } Note the hardcoded "MPI_Info_set(info, "cb_nodes", "1");" and "MPI_Info_set(info, "no_indep_rw", "true");". The above code runs fine when run from multiple cores on Node0. As soon as I add a Node1 to the mix, I get the following error: jpovich@crane mini_test> mpirun -np 2 -hosts crane,node1 /work/jpovich/box_testing/1970/1970_test [mpiexec(a)crane.csi.com] control_cb (pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed [mpiexec(a)crane.csi.com] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)crane.csi.com] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec(a)crane.csi.com] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion The cb_nodes setting seems to have no impact on behavior. I get the same error if I comment out the cb_nodes and no_indep_rw settings. Any help is greatly appreciated, Jon _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Race condition in MPI3 (Fortran, shared memory)
by Min Si 21 Jun '16

21 Jun '16

Hi Christoph, Thanks for the updates. I think this is the correct solution. In short, a=a+1 is not atomic. You have to use an atomic operation in your case, thus it needs to be MPI_ACCUMULATE. Min On 6/21/16 4:04 AM, C.Friedrich wrote: > Thank you. > > After some research I have been able to solve the problem: MPI_WIN_FENCE > is correct, but an assignment like a=a+1 will always lead to race > conditions. This assignment has to be replaced by calls to > MPI_ACCUMULATE, which is "atomic" by design. So, calls to > MPI_ACCUMULATEs do not have to be separated by "fences". However, we > need fences to separate them from the local store (a=0) before and from > the read (e.g., write(*,*) a) afterwards. In the present case, one could > write > > ... > b=1 > tdisp=0 > call mpe_win_fence(a,0,Nwin,err) > do i = 1,10000 > call > mpi_accumulate(b,100,mpi_integer,0,tdisp,100,mpi_integer,mpi_sum,Nwin,err) > > enddo > call mpe_win_fence(a,0,Nwin,err) > ... > > Christoph > > > ------------------------------------------------------------------------------------------------ > > ------------------------------------------------------------------------------------------------ > > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > > ------------------------------------------------------------------------------------------------ > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0