October 2018 - discuss - mailman3.cels.anl.gov

mpich install error (libstdc++.so)
by Zhifeng Yang 18 Oct '18

18 Oct '18

Hi I tried to install mpich using gcc. However, while make, I got the following error message. libtool: link: icpc -shared src/binding/cxx/.libs/initcxx.o -Wl,-rpath,/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/lib/.libs -Wl,-rpath,/cm/shared/apps/gcc/4.8.4/lib/../lib -Wl,-rpath,/home/vy57456/zzbatmos_user/application/gfortran/mpich-3.2.1/lib -Wl,-rpath,/cm/shared/apps/gcc/4.8.4/lib/../lib lib/.libs/libmpi.so -lpthread -L/cm/shared/apps/intel/composer_xe/2015.5.223/ipp/../compiler/lib/intel64 -L/cm/shared/apps/intel/composer_xe/2015.5.223/ipp/lib/intel64 -L/cm/shared/apps/intel/composer_xe/2015.5.223/compiler/lib/intel64 -L/cm/shared/apps/intel/composer_xe/2015.5.223/mkl/lib/intel64 -L/cm/shared/apps/intel/composer_xe/2015.5.223/tbb/lib/intel64/gcc4.4 -L/cm/shared/apps/slurm/14.11.11/lib64/slurm -L/cm/shared/apps/slurm/14.11.11/lib64 -L/usr/cluster/contrib/hdf5/install-1.8.16/lib -L/cm/shared/apps/acml/current/ifort64/lib -L/cm/shared/apps/slurm/14.11.11/lib64/../lib64 -L/cm/shared/apps/slurm/14.11.11/lib64/../lib64/ -L/cm/shared/apps/gcc/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/ -L/cm/shared/apps/gcc/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/../../../../lib64 -L/cm/shared/apps/gcc/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/../../../../lib64/ -L/lib/../lib64 -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/ -L/cm/shared/apps/intel/composer_xe/2015.5.223/ipp/../compiler/lib/intel64/ -L/cm/shared/apps/intel/composer_xe/2015.5.223/ipp/lib/intel64/ -L/cm/shared/apps/intel/composer_xe/2015.5.223/compiler/lib/intel64/ -L/cm/shared/apps/intel/composer_xe/2015.5.223/mkl/lib/intel64/ -L/cm/shared/apps/intel/composer_xe/2015.5.223/tbb/lib/intel64/gcc4.4/ -L/cm/shared/apps/slurm/14.11.11/lib64/slurm/ -L/cm/shared/apps/slurm/14.11.11/lib64/ -L/usr/cluster/contrib/hdf5/install-1.8.16/lib/ -L/cm/shared/apps/acml/current/ifort64/lib/ -L/cm/shared/apps/gcc/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/../../../ -L/lib64 -L/lib/ -L/usr/lib64 -L/usr/lib -limf -lirng -lm -lipgo -ldecimal -lcilkrts /cm/shared/apps/gcc/4.8.4/lib/../lib/libstdc++.so -lintlc -lsvml -lgcc -lgcc_s -lirc_s -ldl -lc -O2 -Wl,-soname -Wl,libmpicxx.so.12 -o lib/.libs/libmpicxx.so.12.1.1 /cm/shared/apps/gcc/4.8.4/lib/../lib/libstdc++.so: could not read symbols: File in wrong format make[2]: *** [lib/libmpicxx.la] Error 1 make[2]: Leaving directory `/umbc/xfs1/zzbatmos/users/vy57456/application/gfortran/source_code/mpich-3.2.1' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/umbc/xfs1/zzbatmos/users/vy57456/application/gfortran/source_code/mpich-3.2.1' make: *** [all] Error 2 Do you have any experience on this error? Thank you Zhifeng _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_Comm_spawn zombies have risen from the dead.
by Thomas Pak 16 Oct '18

16 Oct '18

Dear all, My MPI application spawns a large number of MPI processes using MPI_Comm_spawn over its total lifetime. Unfortunately, I have experienced that this results in zombie processes of hydra_pmi_proxy accumulating over time. As a result, my MPI application will eventually crash because there are too many zombie processes and no new processes can be created. This issue does not seem to be new; I have found references to similar issues in at least three different places: - [mpich-discuss] MPI_Comm_Spawn causing zombies of hydra_pmi_proxy (https://lists.mpich.org/pipermail/discuss/2013-March/000599.html) (March 2013) - [GitHub issue] hydra_pmi_proxy zombie MPI_comm_spawn #1677 (https://github.com/pmodels/mpich/issues/1677) (October 2016) - [Google Groups] Hanging in spawn in master-slave code and hydra_pmi_proxy zombies (https://groups.google.com/forum/#!msg/mpi4py/A9mN-2UkFf8/2gnUEtArDwAJ) (December 2016) In the first reference, a dirty fix was proposed by Silvan Brändli to simply reap all the zombie processes in a loop at the start of the function HYDU_create_process in src/pm/hydra/utils/launch/launch.c. However, I have found that this comes with its own set of problems and makes my MPI application unstable. In the GitHub issue, the problem was marked as "worksforme". However, this problem persists for me and other machines that I have tried to reproduce it on, so it clearly has not yet been resolved. In the Google Groups thread, the problem was "resolved" by avoiding spawning MPI processes entirely, which is not an option in my case. Pavan Balaji mentions in the 2013 mpi-discuss thread that this issue was a problem "once upon a time". It seems to have risen from the dead again. I have attached a short and self-contained program written in C that reproduces the problem. The program simply spawns child processes using MPI_Comm_spawn in an infinite loop, where each child process exits after writing a message to stdout. I use MPI_Comm_disconnect to disconnect the Intercommunicator, but the same problem occurs when using MPI_Comm_free instead. I have also attached the logfiles generated while building MPICH, as described in the README, except for the file mpich-3.2.1/src/pm/hydra/tools/topo/hwloc/hwloc/config.log, as that was missing. I am using MPICH version 3.2.1, which is the latest stable release at the time of writing. Thanks in advance for your help! Best wishes, Thomas Pak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Return error code issues - MPICH 3.2.1
by Ignacio Laguna 10 Oct '18

10 Oct '18

Hi Ken, Thank you, I just submitted an issue (#3366). Note that all the reports are associated with the same function call, MPIR_Setup_intercomm_localcomm, apparently used for intercommunicators, so they essentially refer to the same class of bug, but in different files or communication functions. If at least a simple case could be verified (e.g., bcast) it would be useful for us to mark it a as success case and continue improving the analysis. Thank you!! Ignacio On 10/10/18 2:23 PM, Raffenetti, Kenneth J. wrote: > Hi Ignacio, > > Would you mind creating a Github issue > (https://github.com/pmodels/mpich) to track this? It will take some time > to go through all the locations you have identified, but IMO from the > example you are onto a real issue. > > Ken > > On 10/10/18 10:41 AM, Laguna Peralta, Ignacio wrote: >> Hi MPICH developers, >> >> I would like to report some (unconfirmed) issues on how return error >> codes are propagated within the library (e.g., when using the >> MPI_ERRORS_RETURN handler). >> >> We are developing a static analysis framework that detects the code >> location of such bugs and we have tested it recently in MPICH 3.2.1. The >> framework is giving us a number of reports and we would like to confirm >> with you whether these are real bug cases or just false alarms. >> >> ** What the framework does ** >> >> It analyzes all functions and call paths within the library trying to >> identify cases where a return error code is either: >> >> (a) not saved by the calling function >> (b) saved but later overwritten without MPICH taken an action on the >> error. An action could be, for example, printing a message, aborting, or >> returning the error to the calling function. >> >> An example of case (a) we think we found is in src/mpi/coll/bcast.c:1402: >> >> 1400 /* Get the local intracommunicator */ >> 1401 if (!comm_ptr->local_comm) >> 1402 MPIR_Setup_intercomm_localcomm( comm_ptr ); >> 1403 >> 1404 newcomm_ptr = comm_ptr->local_comm; >> 1405 >> 1406 /* now do the usual broadcast on this intracommunicator >> 1407 with rank 0 as root. */ >> 1408 mpi_errno = MPIR_Bcast_intra(buffer, count, datatype, 0, >> newcomm_ptr, errflag); >> >> In line 1402, the MPIR_Setup_intercomm_localcomm function is called but >> the error code that this function could return is not saved--in line >> 1408 an error code is saved, however, in mpi_errno, which is what we >> would expect for line 1402. >> >> We see that MPICH takes actions such the following when an error is >> observed (from the same file): >> >> 1408 mpi_errno = MPIR_Bcast_intra(buffer, count, datatype, 0, >> newcomm_ptr, errflag); >> 1409 if (mpi_errno) { >> 1410 /* for communication errors, just record the error >> but continue */ >> 1411 *errflag = MPIR_ERR_GET_CLASS(mpi_errno); >> 1412 MPIR_ERR_SET(mpi_errno, *errflag, "**fail"); >> 1413 MPIR_ERR_ADD(mpi_errno_ret, mpi_errno); >> 1414 } >> >> So we took this code and adapted it to line 1402. Then when we inject an >> artificial error code (something different from MPI_SUCCESS) to >> MPIR_Setup_intercomm_localcomm, we see that this fixes the bug and the >> program receives the error when calling MPI_Bcast; otherwise the >> artificially injected error is lost and the program cannot see it. >> >> ** Reports ** >> >> Our framework is reporting the same case in other MPICH 3.2.1 locations: >> >> src/mpi/coll/allgather.c:694 >> src/mpi/coll/allreduce.c:657 >> src/mpi/coll/bcast.c:1402 >> src/mpi/coll/iallgather.c:525 >> src/mpi/coll/iallreduce.c:551 >> src/mpi/coll/iscatter.c:505 >> src/mpi/coll/red_scat_block.c:965 >> src/mpi/coll/scatter.c:509 >> src/mpi/comm/comm_create.c:314 >> src/mpi/comm/comm_split.c:158 >> src/mpi/comm/intercomm_merge.c:301 >> >> ** What we need from you ** >> >> Could someone take a look at these reports and confirm whether they are >> real bug cases (where some action is missing) or they are just false >> reports/alarms since in these locations there is no reason to save the >> error code and take an action? >> >> Thank you for your help. This will help us improve the framework and >> hopefully could report issues to MPICH to make it more reliable. >> >> Thanks! >>

1 0

Re: [mpich-discuss] Return error code issues - MPICH 3.2.1
by Raffenetti, Kenneth J. 10 Oct '18

10 Oct '18

Hi Ignacio, Would you mind creating a Github issue (https://github.com/pmodels/mpich) to track this? It will take some time to go through all the locations you have identified, but IMO from the example you are onto a real issue. Ken On 10/10/18 10:41 AM, Laguna Peralta, Ignacio wrote: > Hi MPICH developers, > > I would like to report some (unconfirmed) issues on how return error > codes are propagated within the library (e.g., when using the > MPI_ERRORS_RETURN handler). > > We are developing a static analysis framework that detects the code > location of such bugs and we have tested it recently in MPICH 3.2.1. The > framework is giving us a number of reports and we would like to confirm > with you whether these are real bug cases or just false alarms. > > ** What the framework does ** > > It analyzes all functions and call paths within the library trying to > identify cases where a return error code is either: > > (a) not saved by the calling function > (b) saved but later overwritten without MPICH taken an action on the > error. An action could be, for example, printing a message, aborting, or > returning the error to the calling function. > > An example of case (a) we think we found is in src/mpi/coll/bcast.c:1402: > > 1400 /* Get the local intracommunicator */ > 1401 if (!comm_ptr->local_comm) > 1402 MPIR_Setup_intercomm_localcomm( comm_ptr ); > 1403 > 1404 newcomm_ptr = comm_ptr->local_comm; > 1405 > 1406 /* now do the usual broadcast on this intracommunicator > 1407 with rank 0 as root. */ > 1408 mpi_errno = MPIR_Bcast_intra(buffer, count, datatype, 0, > newcomm_ptr, errflag); > > In line 1402, the MPIR_Setup_intercomm_localcomm function is called but > the error code that this function could return is not saved--in line > 1408 an error code is saved, however, in mpi_errno, which is what we > would expect for line 1402. > > We see that MPICH takes actions such the following when an error is > observed (from the same file): > > 1408 mpi_errno = MPIR_Bcast_intra(buffer, count, datatype, 0, > newcomm_ptr, errflag); > 1409 if (mpi_errno) { > 1410 /* for communication errors, just record the error > but continue */ > 1411 *errflag = MPIR_ERR_GET_CLASS(mpi_errno); > 1412 MPIR_ERR_SET(mpi_errno, *errflag, "**fail"); > 1413 MPIR_ERR_ADD(mpi_errno_ret, mpi_errno); > 1414 } > > So we took this code and adapted it to line 1402. Then when we inject an > artificial error code (something different from MPI_SUCCESS) to > MPIR_Setup_intercomm_localcomm, we see that this fixes the bug and the > program receives the error when calling MPI_Bcast; otherwise the > artificially injected error is lost and the program cannot see it. > > ** Reports ** > > Our framework is reporting the same case in other MPICH 3.2.1 locations: > > src/mpi/coll/allgather.c:694 > src/mpi/coll/allreduce.c:657 > src/mpi/coll/bcast.c:1402 > src/mpi/coll/iallgather.c:525 > src/mpi/coll/iallreduce.c:551 > src/mpi/coll/iscatter.c:505 > src/mpi/coll/red_scat_block.c:965 > src/mpi/coll/scatter.c:509 > src/mpi/comm/comm_create.c:314 > src/mpi/comm/comm_split.c:158 > src/mpi/comm/intercomm_merge.c:301 > > ** What we need from you ** > > Could someone take a look at these reports and confirm whether they are > real bug cases (where some action is missing) or they are just false > reports/alarms since in these locations there is no reason to save the > error code and take an action? > > Thank you for your help. This will help us improve the framework and > hopefully could report issues to MPICH to make it more reliable. > > Thanks! > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Return error code issues - MPICH 3.2.1
by Laguna Peralta, Ignacio 10 Oct '18

10 Oct '18

Hi MPICH developers, I would like to report some (unconfirmed) issues on how return error codes are propagated within the library (e.g., when using the MPI_ERRORS_RETURN handler). We are developing a static analysis framework that detects the code location of such bugs and we have tested it recently in MPICH 3.2.1. The framework is giving us a number of reports and we would like to confirm with you whether these are real bug cases or just false alarms. ** What the framework does ** It analyzes all functions and call paths within the library trying to identify cases where a return error code is either: (a) not saved by the calling function (b) saved but later overwritten without MPICH taken an action on the error. An action could be, for example, printing a message, aborting, or returning the error to the calling function. An example of case (a) we think we found is in src/mpi/coll/bcast.c:1402: 1400 /* Get the local intracommunicator */ 1401 if (!comm_ptr->local_comm) 1402 MPIR_Setup_intercomm_localcomm( comm_ptr ); 1403 1404 newcomm_ptr = comm_ptr->local_comm; 1405 1406 /* now do the usual broadcast on this intracommunicator 1407 with rank 0 as root. */ 1408 mpi_errno = MPIR_Bcast_intra(buffer, count, datatype, 0, newcomm_ptr, errflag); In line 1402, the MPIR_Setup_intercomm_localcomm function is called but the error code that this function could return is not saved--in line 1408 an error code is saved, however, in mpi_errno, which is what we would expect for line 1402. We see that MPICH takes actions such the following when an error is observed (from the same file): 1408 mpi_errno = MPIR_Bcast_intra(buffer, count, datatype, 0, newcomm_ptr, errflag); 1409 if (mpi_errno) { 1410 /* for communication errors, just record the error but continue */ 1411 *errflag = MPIR_ERR_GET_CLASS(mpi_errno); 1412 MPIR_ERR_SET(mpi_errno, *errflag, "**fail"); 1413 MPIR_ERR_ADD(mpi_errno_ret, mpi_errno); 1414 } So we took this code and adapted it to line 1402. Then when we inject an artificial error code (something different from MPI_SUCCESS) to MPIR_Setup_intercomm_localcomm, we see that this fixes the bug and the program receives the error when calling MPI_Bcast; otherwise the artificially injected error is lost and the program cannot see it. ** Reports ** Our framework is reporting the same case in other MPICH 3.2.1 locations: src/mpi/coll/allgather.c:694 src/mpi/coll/allreduce.c:657 src/mpi/coll/bcast.c:1402 src/mpi/coll/iallgather.c:525 src/mpi/coll/iallreduce.c:551 src/mpi/coll/iscatter.c:505 src/mpi/coll/red_scat_block.c:965 src/mpi/coll/scatter.c:509 src/mpi/comm/comm_create.c:314 src/mpi/comm/comm_split.c:158 src/mpi/comm/intercomm_merge.c:301 ** What we need from you ** Could someone take a look at these reports and confirm whether they are real bug cases (where some action is missing) or they are just false reports/alarms since in these locations there is no reason to save the error code and take an action? Thank you for your help. This will help us improve the framework and hopefully could report issues to MPICH to make it more reliable. Thanks! -- Ignacio Laguna Center for Applied Scientific Computing (CASC) Lawrence Livermore National Laboratory Phone: 925-422-7308, Fax: 925-422-6287 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Assertion failed in file ch3i_comm.c
by Chao-Chin Yang 09 Oct '18

09 Oct '18

Hi Min, Thanks for the prompt reply. I have tested 3.3b3 and indeed the assertion failure is gone. Sincerely, Chao-Chin Yang Postdoctoral Scholar Department of Physics and Astronomy University of Nevada, Las Vegas, U.S.A. > On Oct 8, 2018, at 7:07 AM, Si, Min <msi(a)anl.gov> wrote: > > Hi Chao-Chin, > > The code has been removed in commit 61c2f6f6ba90ca0c9a988fedf714f2f606674121 > Author: Wesley Bland <wesley.bland(a)intel.com> <mailto:[email protected]> > Date: Wed Nov 15 10:00:22 2017 -0600 > > ch3: Remove unneeded internal barrier functions > > Signed-off-by: Pavan Balaji <balaji(a)anl.gov> <mailto:[email protected]> > > > Could you please try the latest MPICH 3.3b3 tarball or clone the MPICH master branch at https://github.com/pmodels/mpich <https://github.com/pmodels/mpich> ? > > If you still see any failure, please send us the config.log file at your MPICH build directory, and a simple reproducer program. > > Thanks, > Min > On 2018/10/06 22:59, Chao-Chin Yang wrote: >> Hi, >> >> I am using >> >> mpicxx for MPICH version 3.2.1 >> Apple LLVM version 10.0.0 (clang-1000.11.45.2) >> Target: x86_64-apple-darwin17.7.0 >> Thread model: posix >> InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin >> >> But I run into a runtime error: >> >> Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3i_comm.c at line 139: 0 >> internal ABORT - process 0 >> Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3i_comm.c at line 139: 0 >> internal ABORT - process 1 >> >> I looked into the source code ch3i_comm.c and found that there is a FIXME comment but I am not quite sure I get the picture. So my question is: >> >> In what condition(s) does one get this assertion error? >> >> Sincerely, >> >> Chao-Chin Yang >> Postdoctoral Scholar >> Department of Physics and Astronomy >> University of Nevada, Las Vegas, U.S.A. >> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] couldn't built mpich-master on SuSE Linux with Sun C
by Raffenetti, Kenneth J. 09 Oct '18

09 Oct '18

Siegmar, In case you didn't see it, this is the PR: https://github.com/pmodels/mpich/pull/3353. Ken On 10/3/18 3:53 PM, Congiu, Giuseppe wrote: > Siegmar, > > I have created a pull request that should fix the fortran detection > problem with mpich. Can you try it and see if it works for you? > > Thank you, > Giuseppe > > > >> On Oct 3, 2018, at 1:25 PM, Congiu, Giuseppe <gcongiu(a)anl.gov >> <mailto:[email protected]>> wrote: >> >> Hi Siegmar, >> >> We have looked into the problem. It seems this is not MPICH specific >> but rather libtools specific. I have created an issue >> (https://github.com/pmodels/mpich/issues/3350) for this. You can look >> at it for bug fix updates. >> >> Best, >> Giuseppe >> >>> On Oct 1, 2018, at 6:16 AM, Siegmar Gross >>> <siegmar.gross(a)informatik.hs-fulda.de >>> <mailto:[email protected]>> wrote: >>> >>> Hi Giuseppe, >>> >>> my email is being held until the list moderator can review it for >>> approval, >>> because the attachments are too big. Therefore, I send it directly to >>> you. >>> >>> >>> Best regards >>> >>> Siegmar >>> >>> >>> -------- Forwarded Message -------- >>> Subject: Re: [mpich-discuss] couldn't built mpich-master on SuSE >>> Linux with Sun C >>> Date: Mon, 1 Oct 2018 10:37:58 +0200 >>> From: Siegmar Gross <Siegmar.Gross(a)informatik.hs-fulda.de >>> <mailto:[email protected]>> >>> To: Users MPICH <discuss(a)mpich.org <mailto:[email protected]>> >>> >>> Hi Giuseppe, >>> >>> today I've tried to build the latest master version so that you don't >>> have >>> to deal with an old version. I still get the same error. >>> >>> loki mpich-master_2018-10-01_Linux.x86_64.64_cc 150 cc -V >>> cc: Studio 12.6 Sun C 5.15 Linux_i386 2017/05/30 >>> >>> I've attached the requested files. Thank you very much for any help >>> in advance. >>> >>> >>> Best regards >>> >>> Siegmar >>> >>> >>> >>> On 09/26/18 21:41, Congiu, Giuseppe wrote: >>>> Hi Siegmar, >>>> I have tried replicating the linking problem using the configure >>>> options you provided. I have used Oracle Developer Studio 12.3. >>>> Mpich master “701bc7e5f84467188dcdc1ff585349e994a9990a". Compilation >>>> worked fine for me. Can you attach your config.log >>>> file so I can have a look at it? Moreover, can you recompile Mpich >>>> with "make V=1" and attach the corresponding output as well? >>>> Thank you, >>>> Giuseppe >>>>> On Jun 6, 2018, at 6:21 AM, Siegmar Gross >>>>> <siegmar.gross(a)informatik.hs-fulda.de >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I tried to build mpich-master on my "SUSE Linux Enterprise Server 12.3 >>>>> (x86_64)" with Sun C 5.15 (Oracle Developer Studio 12.6). >>>>> Unfortunately, >>>>> I get the following error, that I already reported already on May 9th. >>>>> >>>>> >>>>> loki mpich-master_2018-06-06_Linux.x86_64.64_cc 122 tail -22 >>>>> log.make.Linux.x86_64.64_cc | grep -v NOTICE >>>>> "../mpich/src/include/mpir_handlemem.h", line 61: warning: null >>>>> dimension: indirect >>>>> "../mpich/src/include/mpir_handlemem.h", line 122: warning: null >>>>> dimension: indirect >>>>> MOD src/binding/fortran/use_mpi/mpi.mod-stamp >>>>> "./src/include/mpichconf.h", line 1708: Warning (Anachronism): >>>>> Attempt to redefine __restrict__ without using #undef. >>>>> 1 Warning(s) detected. >>>>> f90: Warning: Option -shared passed to ld, if ld is invoked, >>>>> ignored otherwise >>>>> f90: Warning: Option --whole-archive passed to ld, if ld is >>>>> invoked, ignored otherwise >>>>> f90: Warning: Option --no-whole-archive passed to ld, if ld is >>>>> invoked, ignored otherwise >>>>> f90: Warning: Option -path passed to ld, if ld is invoked, ignored >>>>> otherwise >>>>> f90: Warning: Option -path passed to ld, if ld is invoked, ignored >>>>> otherwise >>>>> f90: Warning: Option -soname passed to ld, if ld is invoked, >>>>> ignored otherwise >>>>> /usr/bin/ld: -r and -shared may not be used together >>>>> postopt: error: ld failed to link the binary >>>>> Makefile:16787: recipe for target 'lib/libmpi.la >>>>> <http://libmpi.la/>' failed >>>>> make[2]: *** [lib/libmpi.la <http://libmpi.la/>] Error 2 >>>>> make[2]: Leaving directory >>>>> '/export2/src/mpich-master/mpich-master_2018-06-06_Linux.x86_64.64_cc' >>>>> Makefile:39797: recipe for target 'all-recursive' failed >>>>> make[1]: *** [all-recursive] Error 1 >>>>> make[1]: Leaving directory >>>>> '/export2/src/mpich-master/mpich-master_2018-06-06_Linux.x86_64.64_cc' >>>>> Makefile:10453: recipe for target 'all' failed >>>>> make: *** [all] Error 2 >>>>> loki mpich-master_2018-06-06_Linux.x86_64.64_cc 123 >>>>> >>>>> >>>>> I used the following commands to update and configure MPICH. >>>>> >>>>> cd mpich >>>>> git pull https://github.com/pmodels/mpich.git master >>>>> git submodule update --init --recursive >>>>> autogen.sh >>>>> >>>>> mkdir mpich-master_`date -I'date'`_${SYSTEM_ENV}.${MACHINE_ENV}.64_cc >>>>> cd mpich-master_`date -I'date'`_${SYSTEM_ENV}.${MACHINE_ENV}.64_cc >>>>> >>>>> ../mpich/configure \ >>>>> --prefix=/usr/local/mpich-master_64_cc \ >>>>> --libdir=/usr/local/mpich-master_64_cc/lib64 \ >>>>> --includedir=/usr/local/mpich-master_64_cc/include64 \ >>>>> CC="cc" CXX="CC" F77="f77" FC="f95" \ >>>>> CFLAGS="-m64" CXXFLAGS="-m64" FFLAGS="-m64" FCFLAGS="-m64" \ >>>>> LDFLAGS="-m64 -L/usr/lib64 -R/usr/lib64" \ >>>>> --enable-fortran=yes --enable-cxx --enable-romio \ >>>>> --enable-debuginfo --enable-smpcoll \ >>>>> --enable-threads=multiple --with-thread-package=posix \ >>>>> --enable-shared \ >>>>> |& tee log.configure.$SYSTEM_ENV.$MACHINE_ENV.64_cc >>>>> >>>>> make -j 6 |& tee log.make.$SYSTEM_ENV.$MACHINE_ENV.64_cc >>>>> >>>>> >>>>> I would be grateful, if somebody can fix the problem or is it a >>>>> problem of the >>>>> Oracle compiler?. Do you need anything else? Thank you very much >>>>> for any help >>>>> in advance. >>>>> >>>>> >>>>> Kind regards >>>>> >>>>> Siegmar >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> <config.log><log.configure.Linux.x86_64.64_cc><log.make.Linux.x86_64.64_cc> >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH-3.2.1 crashes on MPI_BARRIER call
by Congiu, Giuseppe 09 Oct '18

09 Oct '18

Hello Bill, This seems a network connection problem. Please make sure that every machine can connect to every other machine and there is no firewall blocking connections. Best, Giuseppe > On Jun 13, 2018, at 9:24 AM, Bill Petrick <wpetrick(a)industrialimaging.com> wrote: > > Hi all, > > For the past 9 years I've been running MPICH-1.2.7 with the g95 fortran compiler. I just upgraded to MPICH-3.2.1 with a gfortran compiler. My first tests running on 2 nodes were fine. When I ran the program on 3 or more nodes it crashes at the first MPI_BARRIER call. As a test I ran the following "helloworld" program. It runs on all nodes just fine until I add the MPI_BARRIER statement. With the MPI_BARRIER statement it also runs fine on 2 nodes but crashes when run on 3 or more nodes. I execute with the following statement. > > mpiexec -f machines -launcher rsh –n 2 ./helloworld > > This program runs fine on all nodes without the MPI_BARRIER call but crashes on more than two nodes with the BARRIER call. > ____________________________________________________ > program main > use mpi > integer ( kind = 4 ) error > integer ( kind = 4 ) id > integer ( kind = 4 ) p > real ( kind = 8 ) wtime > ! Initialize MPI. > call MPI_Init ( error ) > ! Get the number of processes. > call MPI_Comm_size ( MPI_COMM_WORLD, p, error ) > ! Get the individual process ID. > call MPI_Comm_rank ( MPI_COMM_WORLD, id, error ) > call MPI_BARRIER(MPI_COMM_WORLD,error) > ! > ! Print a message. > ! > .................etc__________________________________________ > > However when I run it on 3 or more nodes it crashes at MPI_BARRIER with the following error message > __________________________________________________________________________ > iic@node0:/clusterfiles$ mpiexec -f machines -launcher rsh -n 3 ./helloworld > Fatal error in PMPI_Barrier: Unknown error class, error stack: > PMPI_Barrier(425)......: MPI_Barrier(MPI_COMM_WORLD) failed > MPIR_Barrier_impl(332).: Failure during collective > MPIR_Barrier_impl(327).: > MPIR_Barrier(292)......: > MPIR_Barrier_intra(180): Failure during collective > Fatal error in PMPI_Barrier: Unknown error class, error stack: > PMPI_Barrier(425)...............: MPI_Barrier(MPI_COMM_WORLD) failed > MPIR_Barrier_impl(332)..........: Failure during collective > MPIR_Barrier_impl(327)..........: > MPIR_Barrier(292)...............: > MPIR_Barrier_intra(169).........: > MPID_nem_tcp_connpoll(1845).....: Communication error with rank 2: Connection refused > MPIR_Barrier_intra(169).........: > MPIDI_CH3U_Recvq_FDU_or_AEP(629): Communication error with rank 2 > iic@node0:/clusterfiles$ ______________________________________________________ > > > /etc/hosts is consistent across all nodes. > /usr/bin/rsh exists on all nodes > the "machines" file is the same on all nodes. > > Anyone have any ideas about what could be wrong? Thanks, Bill > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

parallel execution error
by Haider Abbas 08 Oct '18

08 Oct '18

Dear all, Parallel execution of 'gamess' give the output which is given below. I guess that this is due to some internal firewall of ubuntu because I am using a separate switch with no Internet connection. could you please let me know how to overcome this problem. which command about iptables should I run on master and other nodes. As I am first time preparing the cluster so guide me about the running the cluster, from which node should I start my job, master or some other nodes. with regards yours sincerely Haider Abbas ----- GAMESS execution script 'rungms' ----- This job is running on host physics-OptiPlex-3046 under operating system Linux at Sat Oct 6 14:32:01 IST 2018 Available scratch disk space (Kbyte units) at beginning of the job is Filesystem 1K-blocks Used Available Use% Mounted on master:/home/mpiuser 953145344 23300096 881404928 3% /home/mpiuser GAMESS temporary binary files will be written to /home/mpiuser/gamess GAMESS supplementary output files will be written to /home/mpiuser/scr Copying input file siguanine.inp to your run's scratch directory... cp siguanine.inp /home/mpiuser/gamess/siguanine.F05 unset echo /home/mpiuser/gamess/ddikick.x /home/mpiuser/gamess/gamess.00.x siguanine -ddi 10 40 master:cpus=4 node1:cpus=4 node2:cpus=4 node3:cpus=4 node4:cpus=4 node5:cpus=4 node6:cpus=4 node7:cpus=4 node8:cpus=4 node9:cpus=4 -scr /home/mpiuser/gamess Distributed Data Interface kickoff program. Initiating 40 compute processes on 10 nodes to run the following command: /home/mpiuser/gamess/gamess.00.x siguanine TCP connect error: ECONNREFUSED. DDI Process 40: error code 911 TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. DDI Process 42: error code 911 TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. DDI Process 1: error code 911 TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. DDI Process 41: error code 911 TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. DDI Process 43: error code 911 TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. DDI Process 2: error code 911 TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. DDI Process 3: error code 911 TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. DDI Process 0: error code 911 TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. TCP connect error: ECONNREFUSED. TCP: Connect failed. physics-OptiPlex-3046 -> master:35850. ddikick.x: Timed out while waiting for DDI processes to check in. ddikick.x: Fatal error detected. The error is most likely to be in the application, so check for input errors, disk space, memory needs, application bugs, etc. ddikick.x will now clean up all processes, and exit... ddikick.x: Sending kill signal to DDI processes. ddikick.x: Execution terminated due to error(s). unset echo ----- accounting info ----- Files used on the master node physics-OptiPlex-3046 were: -rw-r--r-- 1 mpiuser mpiuser 2594 Oct 6 14:28 /home/mpiuser/gamess/siguanine.F05 -rw-r--r-- 1 physics physics 2594 Aug 8 2016 /home/mpiuser/gamess/siguanine.inp -rw-rw-r-- 1 mpiuser mpiuser 7856 Oct 6 14:29 /home/mpiuser/gamess/siguanine.out ls: No match. ls: No match. ls: No match. Files from node1 are: -rw-r--r-- 1 physics physics 2594 Aug 8 2016 /home/mpiuser/gamess/siguanine.inp -rw-rw-r-- 1 mpiuser mpiuser 8166 Oct 6 14:29 /home/mpiuser/gamess/siguanine.out Files from node2 are: -rw-r--r-- 1 physics physics 2594 Aug 8 2016 /home/mpiuser/gamess/siguanine.inp -rw-rw-r-- 1 mpiuser mpiuser 8352 Oct 6 14:29 /home/mpiuser/gamess/siguanine.out Files from node3 are: -rw-r--r-- 1 physics physics 2594 Aug 8 2016 /home/mpiuser/gamess/siguanine.inp -rw-rw-r-- 1 mpiuser mpiuser 8538 Oct 6 14:29 /home/mpiuser/gamess/siguanine.out Files from node4 are: _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Assertion failed in file ch3i_comm.c
by Si, Min 08 Oct '18

08 Oct '18

Hi Chao-Chin, The code has been removed in commit 61c2f6f6ba90ca0c9a988fedf714f2f606674121 Author: Wesley Bland <wesley.bland(a)intel.com><mailto:[email protected]> Date: Wed Nov 15 10:00:22 2017 -0600 ch3: Remove unneeded internal barrier functions Signed-off-by: Pavan Balaji <balaji(a)anl.gov><mailto:[email protected]> Could you please try the latest MPICH 3.3b3 tarball or clone the MPICH master branch at https://github.com/pmodels/mpich ? If you still see any failure, please send us the config.log file at your MPICH build directory, and a simple reproducer program. Thanks, Min On 2018/10/06 22:59, Chao-Chin Yang wrote: Hi, I am using mpicxx for MPICH version 3.2.1 Apple LLVM version 10.0.0 (clang-1000.11.45.2) Target: x86_64-apple-darwin17.7.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin But I run into a runtime error: Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3i_comm.c at line 139: 0 internal ABORT - process 0 Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3i_comm.c at line 139: 0 internal ABORT - process 1 I looked into the source code ch3i_comm.c and found that there is a FIXME comment but I am not quite sure I get the picture. So my question is: In what condition(s) does one get this assertion error? Sincerely, Chao-Chin Yang Postdoctoral Scholar Department of Physics and Astronomy University of Nevada, Las Vegas, U.S.A. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0