March 2018 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_LOOKUP_NAME getting wrong port back?
by Alexander Rast 21 Mar '18

21 Mar '18

Fixed. At least the nameserver lookup part of it. Those [cli_1] messages remain but the application starts up and runs to completion without errors on both sides. The solution is to call MPI_Unpublish_name on the 'server' node that published the name. It's not immediately obvious from the MPI documentation that you need to do this, and I get the impression from several on-line posts in various groups it's not immediately obvious to others, either, but it makes sense. Anyway, maybe this note will be useful for posterity. So now we're just down to those mysterious error messages from what looks to be the nameserver. On Wed, Mar 21, 2018 at 2:10 PM, Alexander Rast < alex.rast.technical(a)gmail.com> wrote: > More information: this looks as if it may be a bug in hydra_nameserver. I > found the following: > > If you start up the hydra_nameserver, and do an mpiexec with both my > 'server' and 'client' groups, *the first time*, if you've got things set up > OK, the client retrieves the right port and connects. > However, if when the MPI application exits, you then try to run it a > second time on both 'server' and 'client', the server gets a new port ID > and appears to store it to the nameserver, BUT the client retrieves a port > ID that is basically, the port ID the server had on its first run, plus > some (nonrandom) garbage at the end. The garbage at the end appears to be > some small slice of one of the text strings corresponding to a port ID. It > seems therefore that the nameserver isn't (entirely) deleting the published > port information when an MPI application exits. And then it's doing > something strange when it tries to look up the same service name for a > different run. If you kill the hydra_nameserver process, then when you run > the applications again with 'server' and 'client' they connect. > > Incidentally, I got the following errors reported from what looks to be > the hydra_nameserver on one run, when the 'server' group executed > MPI_COMM_ACCEPT: > > [cli_1]: PMIU_parse_keyvals: unexpected key delimiter at character 96 in > cmd > [cli_1]: parse_keyvals failed -1 > > However, the 'client' group successfully connected in spite of those > messages. > > On Wed, Mar 21, 2018 at 12:58 AM, Alexander Rast < > alex.rast.technical(a)gmail.com> wrote: > >> I'm trying to test the MPI publish-connect methods and running into a >> situation where for some reason MPI_LOOKUP_NAME is returning the wrong >> port. You can see the test source and output in the attached zipfile. >> >> I'm testing this (on Ubuntu 16.04, MPICH 3.2) by opening 2 terminal >> windows and starting one group of 2 processes (the 'server' group) in one, >> and another group (the 'client' group) in another. Beforehand I've started >> the hydra name server using hydra_nameserver & (in the same terminal >> session as the server group) The exact command line for the server group is: >> >> mpiexec -nameserver DeepThought -n 1 ./MPI_Example_C_0 server : -n 1 >> ./MPI_Example_C_1 server >> >> and for the client group: >> >> mpiexec -nameserver DeepThought -n 1 ./MPI_Example_C_0 client : -n 1 >> ./MPI_Example_C_1 client >> >> If you look at the attached output files, you can see that the problem is >> straightforward: MPI_LOOKUP_NAME is returning the wrong port name for the >> server group. Rather surprisingly, too, it *always* returns the exact same >> port name for any run: >> >> tag#0$description#DeepThought$port#50178$ifname#127.0.1.1$127.0.P >> >> even though the port name reported for the server varies: in this run it >> was >> >> tag#0$description#DeepThought$port#33341$ifname#127.0.1.1$ >> >> The clients are clearly performing the lookup and successfully querying >> the nameserver because if I don't run the server and simply run the clients >> they abort saying they couldn't find a published port, as expected. >> >> So something strange is going on. Has anyone run into this? Is there >> something I need to do otherwise? >> >> Thanks for any help. >> >> Alex Rast >> > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_LOOKUP_NAME getting wrong port back?
by Alexander Rast 21 Mar '18

21 Mar '18

More information: this looks as if it may be a bug in hydra_nameserver. I found the following: If you start up the hydra_nameserver, and do an mpiexec with both my 'server' and 'client' groups, *the first time*, if you've got things set up OK, the client retrieves the right port and connects. However, if when the MPI application exits, you then try to run it a second time on both 'server' and 'client', the server gets a new port ID and appears to store it to the nameserver, BUT the client retrieves a port ID that is basically, the port ID the server had on its first run, plus some (nonrandom) garbage at the end. The garbage at the end appears to be some small slice of one of the text strings corresponding to a port ID. It seems therefore that the nameserver isn't (entirely) deleting the published port information when an MPI application exits. And then it's doing something strange when it tries to look up the same service name for a different run. If you kill the hydra_nameserver process, then when you run the applications again with 'server' and 'client' they connect. Incidentally, I got the following errors reported from what looks to be the hydra_nameserver on one run, when the 'server' group executed MPI_COMM_ACCEPT: [cli_1]: PMIU_parse_keyvals: unexpected key delimiter at character 96 in cmd [cli_1]: parse_keyvals failed -1 However, the 'client' group successfully connected in spite of those messages. On Wed, Mar 21, 2018 at 12:58 AM, Alexander Rast < alex.rast.technical(a)gmail.com> wrote: > I'm trying to test the MPI publish-connect methods and running into a > situation where for some reason MPI_LOOKUP_NAME is returning the wrong > port. You can see the test source and output in the attached zipfile. > > I'm testing this (on Ubuntu 16.04, MPICH 3.2) by opening 2 terminal > windows and starting one group of 2 processes (the 'server' group) in one, > and another group (the 'client' group) in another. Beforehand I've started > the hydra name server using hydra_nameserver & (in the same terminal > session as the server group) The exact command line for the server group is: > > mpiexec -nameserver DeepThought -n 1 ./MPI_Example_C_0 server : -n 1 > ./MPI_Example_C_1 server > > and for the client group: > > mpiexec -nameserver DeepThought -n 1 ./MPI_Example_C_0 client : -n 1 > ./MPI_Example_C_1 client > > If you look at the attached output files, you can see that the problem is > straightforward: MPI_LOOKUP_NAME is returning the wrong port name for the > server group. Rather surprisingly, too, it *always* returns the exact same > port name for any run: > > tag#0$description#DeepThought$port#50178$ifname#127.0.1.1$127.0.P > > even though the port name reported for the server varies: in this run it > was > > tag#0$description#DeepThought$port#33341$ifname#127.0.1.1$ > > The clients are clearly performing the lookup and successfully querying > the nameserver because if I don't run the server and simply run the clients > they abort saying they couldn't find a published port, as expected. > > So something strange is going on. Has anyone run into this? Is there > something I need to do otherwise? > > Thanks for any help. > > Alex Rast > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_LOOKUP_NAME getting wrong port back?
by Alexander Rast 20 Mar '18

20 Mar '18

I'm trying to test the MPI publish-connect methods and running into a situation where for some reason MPI_LOOKUP_NAME is returning the wrong port. You can see the test source and output in the attached zipfile. I'm testing this (on Ubuntu 16.04, MPICH 3.2) by opening 2 terminal windows and starting one group of 2 processes (the 'server' group) in one, and another group (the 'client' group) in another. Beforehand I've started the hydra name server using hydra_nameserver & (in the same terminal session as the server group) The exact command line for the server group is: mpiexec -nameserver DeepThought -n 1 ./MPI_Example_C_0 server : -n 1 ./MPI_Example_C_1 server and for the client group: mpiexec -nameserver DeepThought -n 1 ./MPI_Example_C_0 client : -n 1 ./MPI_Example_C_1 client If you look at the attached output files, you can see that the problem is straightforward: MPI_LOOKUP_NAME is returning the wrong port name for the server group. Rather surprisingly, too, it *always* returns the exact same port name for any run: tag#0$description#DeepThought$port#50178$ifname#127.0.1.1$127.0.P even though the port name reported for the server varies: in this run it was tag#0$description#DeepThought$port#33341$ifname#127.0.1.1$ The clients are clearly performing the lookup and successfully querying the nameserver because if I don't run the server and simply run the clients they abort saying they couldn't find a published port, as expected. So something strange is going on. Has anyone run into this? Is there something I need to do otherwise? Thanks for any help. Alex Rast _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] missing mpi_f08 module in MPICH 3.2
by Thomas Jahns 18 Mar '18

18 Mar '18

Hello Jeff, > On Mar 15, 2018, at 23:27 , Jeff Hammond <jeff.science(a)gmail.com> wrote: > /opt/gcc$ find git HEAD -name ISO_Fortran_binding.h | echo MISSING > MISSING while I don’t know about what gfortran 8 provides, that is probably not the command you are looking for but rather: find git HEAD -name ISO_Fortran_binding.h -print or find git HEAD -name ISO_Fortran_binding.h || echo MISSING Regards, Thomas _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] missing mpi_f08 module in MPICH 3.2
by Damian Rouson 15 Mar '18

15 Mar '18

On March 15, 2018 at 3:25:00 PM, Jeff Hammond (jeff.science(a)gmail.com) wrote: This is the MPICH configure error: mcve.c:1:10: fatal error: ISO_Fortran_binding.h: No such file or directory #include <ISO_Fortran_binding.h> ^~~~~~~~~~~~~~~~~~~~~~~ compilation terminated. Thanks! This helps a ton. Relevant GCC content: https://gcc.gnu.org/bugzilla/show_bug.cgi?format=multiple&id=48820 https://gcc.gnu.org/ml/fortran/2012-07/msg00115.html Fascinating! I don't understand why GCC hasn't implemented this header in the 5+ years that they've known that MPI needs it. The person who created the above content was a prolific contributor whom we lost to a sailing hobby a couple of years ago. The other issue is chronological. GCC 8 will be the first version that is fully Fortran 95 compliant. It will also be the first that is Fortran 2003 compliant. And it comes quite close to full Fortran 2008 compliance. The missing iso_fortran_binding.h is officially a Fortran 2018 feature, which is probably why it hasn’t risen to the top of a gfortran developer's list, but the good news is GCC 8 will also support a very sizable chunk of the new Fortran 2018 features too so I imagine this will float to the top of someone’s list after GCC 8 releases and the trunk is open for new features. There is a graduate student coming to work with me for 11 weeks this Spring who might be interested in working on this. It won’t be the first thing he does, but there’s a good chance he’ll at least get started with it before he leaves and it’s really helpful to have the above links as starting point. Damian _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] missing mpi_f08 module in MPICH 3.2
by Jeff Hammond 15 Mar '18

15 Mar '18

So GCC had this header in 2013 ( https://gcc.gnu.org/viewcvs/gcc/branches/fortran-dev/libgfortran/ISO_Fortra…) but it does not appear anywhere in the latest GCC source or install. /opt/gcc$ find git HEAD -name ISO_Fortran_binding.h | echo MISSING MISSING Jeff On Thu, Mar 15, 2018 at 3:24 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > This is the MPICH configure error: > > configure:44317: gcc-8 -c -O2 -I/home/jrhammon/Work/MPI/MPICH/git/build/src/mpl/include -I/home/jrhammon/Work/MPI/MPICH/git/src/mpl/include -I/home/jrhammon/Work/MPI/MPICH/git/src/openpa/src -I/home/jrhammon/Work/MPI/MPICH/git/build/src/openpa/src -D_REENTRANT -I/home/jrhammon/Work/MPI/MPICH/git/build/src/mpi/romio/include conftest.c >&5 > conftest.c:155:10: fatal error: ISO_Fortran_binding.h: No such file or directory > #include <ISO_Fortran_binding.h> > ^~~~~~~~~~~~~~~~~~~~~~~ > > It has a trivial reproducer: > > $ cat mcve.c > #include <ISO_Fortran_binding.h> > > int main() > { > return 0; > } > > $ gcc-8 -c mcve.c > mcve.c:1:10: fatal error: ISO_Fortran_binding.h: No such file or directory > #include <ISO_Fortran_binding.h> > ^~~~~~~~~~~~~~~~~~~~~~~ > compilation terminated. > > Relevant GCC content: > > https://gcc.gnu.org/bugzilla/show_bug.cgi?format=multiple&id=48820 > https://gcc.gnu.org/ml/fortran/2012-07/msg00115.html > > I don't understand why GCC hasn't implemented this header in the 5+ years that they've known that MPI needs it. > > Jeff > > On Thu, Mar 15, 2018 at 1:52 PM, Damian Rouson < damian(a)sourceryinstitute.org> wrote: >> >> On March 15, 2018 at 1:05:34 PM, Kenneth Raffenetti (raffenet(a)mcs.anl.gov) wrote: >> >> F08 bindings will build if they pass the included compiler tests during >> configure. Unfortunately gfortran is unable to build the binding. >> >> Thanks for the quick reply. Is there a GCC bug report that demonstrates the issue? The gfortran trunk now supports most of the required Fortran 2008 + TS29113 features so I’d like to see what’s missing and add myself to the cc list on the bug report. >> >> The >> latest Intel compiler suite will work, if you have access to that. >> >> Thanks. I do. >> >> Damian >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] missing mpi_f08 module in MPICH 3.2
by Jeff Hammond 15 Mar '18

15 Mar '18

This is the MPICH configure error: configure:44317: gcc-8 -c -O2 -I/home/jrhammon/Work/MPI/MPICH/git/build/src/mpl/include -I/home/jrhammon/Work/MPI/MPICH/git/src/mpl/include -I/home/jrhammon/Work/MPI/MPICH/git/src/openpa/src -I/home/jrhammon/Work/MPI/MPICH/git/build/src/openpa/src -D_REENTRANT -I/home/jrhammon/Work/MPI/MPICH/git/build/src/mpi/romio/include conftest.c >&5 conftest.c:155:10: fatal error: ISO_Fortran_binding.h: No such file or directory #include <ISO_Fortran_binding.h> ^~~~~~~~~~~~~~~~~~~~~~~ It has a trivial reproducer: $ cat mcve.c #include <ISO_Fortran_binding.h> int main() { return 0; } $ gcc-8 -c mcve.c mcve.c:1:10: fatal error: ISO_Fortran_binding.h: No such file or directory #include <ISO_Fortran_binding.h> ^~~~~~~~~~~~~~~~~~~~~~~ compilation terminated. Relevant GCC content: https://gcc.gnu.org/bugzilla/show_bug.cgi?format=multiple&id=48820 https://gcc.gnu.org/ml/fortran/2012-07/msg00115.html I don't understand why GCC hasn't implemented this header in the 5+ years that they've known that MPI needs it. Jeff On Thu, Mar 15, 2018 at 1:52 PM, Damian Rouson <damian(a)sourceryinstitute.org > wrote: > On March 15, 2018 at 1:05:34 PM, Kenneth Raffenetti (raffenet(a)mcs.anl.gov) > wrote: > > F08 bindings will build if they pass the included compiler tests during > configure. Unfortunately gfortran is unable to build the binding. > > Thanks for the quick reply. Is there a GCC bug report that demonstrates > the issue? The gfortran trunk now supports most of the required Fortran > 2008 + TS29113 features so I’d like to see what’s missing and add myself to > the cc list on the bug report. > > The > latest Intel compiler suite will work, if you have access to that. > > Thanks. I do. > > Damian > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] missing mpi_f08 module in MPICH 3.2
by Damian Rouson 15 Mar '18

15 Mar '18

On March 15, 2018 at 1:05:34 PM, Kenneth Raffenetti (raffenet(a)mcs.anl.gov) wrote: F08 bindings will build if they pass the included compiler tests during configure. Unfortunately gfortran is unable to build the binding. Thanks for the quick reply. Is there a GCC bug report that demonstrates the issue? The gfortran trunk now supports most of the required Fortran 2008 + TS29113 features so I’d like to see what’s missing and add myself to the cc list on the bug report. The latest Intel compiler suite will work, if you have access to that. Thanks. I do. Damian

1 0

Re: [mpich-discuss] missing mpi_f08 module in MPICH 3.2
by Kenneth Raffenetti 15 Mar '18

15 Mar '18

F08 bindings will build if they pass the included compiler tests during configure. Unfortunately gfortran is unable to build the binding. The latest Intel compiler suite will work, if you have access to that. Ken On 03/15/2018 02:29 PM, Damian Rouson wrote: > > According to the MPICH Changes web page, "Fortran 2008 bindings are enabled by default and fully supported” starting in MPICH 3.2. Could someone please point me to documentation on using the bindings? My installation of MPICH 3.2 doesn’t find the mpi_f08 module described in MPI 3.1 standard. My draft copy of the standard states the following at the beginning of Section 17.1.2: "An MPI implementation providing a Fortran interface must provide a module named mpi_f08 that can be used in a Fortran program.” Below is an example of what I’m seeing. > > Damian > > > $ cat use_mpi_f08.f90 > program main > use mpi_f08 > implicit none > end program > > $ mpifort use_mpi_f08.f90 > use_mpi_f08.f90:2:6: > > use mpi_f08 > 1 > Fatal Error: Can't open module file ‘mpi_f08.mod’ for reading at (1): No such file or directory > compilation terminated. > > $ mpiexec --version > HYDRA build details: > Version: 3.2 > Release Date: Wed Nov 11 22:06:48 CST 2015 > CC: gcc > CXX: g++ > F77: gfortran > F90: gfortran > Configure options: '--disable-option-checking' '--prefix=/opt/mpich/3.2/gnu/7.2.0' 'CC=gcc' 'CXX=g++' 'FC=gfortran' '--cache-file=/dev/null' '--srcdir=/home/rouson/Desktop/Builds/opencoarrays/prerequisites/downloads/mpich-3.2/src/pm/hydra' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=-lpthread ' 'CPPFLAGS= -I/home/rouson/Desktop/Builds/opencoarrays/prerequisites/builds/mpich-3.2/src/mpl/include -I/home/rouson/Desktop/Builds/opencoarrays/prerequisites/downloads/mpich-3.2/src/mpl/include -I/home/rouson/Desktop/Builds/opencoarrays/prerequisites/downloads/mpich-3.2/src/openpa/src -I/home/rouson/Desktop/Builds/opencoarrays/prerequisites/builds/mpich-3.2/src/openpa/src -D_REENTRANT -I/home/rouson/Desktop/Builds/opencoarrays/prerequisites/builds/mpich-3.2/src/mpi/romio/include' > Process Manager: pmi > Launchers available: ssh rsh fork slurm ll lsf sge manual persist > Topology libraries available: hwloc > Resource management kernels available: user slurm ll lsf sge pbs cobalt > Checkpointing libraries available: blcr > Demux engines available: poll select > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Missing instructions
by Gropp, William D 15 Mar '18

15 Mar '18

Thanks, Ken. That got me through the autogen.sh . Bill William Gropp Director and Chief Scientist, NCSA Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign > On Mar 15, 2018, at 2:36 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > > I updated this section to include a command to retrieve the needed submodules. > > https://wiki.mpich.org/mpich/index.php/Getting_And_Building_MPICH#Setting_u… > > Ken > > On 03/15/2018 07:49 AM, William Gropp wrote: >> When I run autogen.sh, I get: >> gropp$ ./autogen.sh >> #################################### >> ## Checking user environment >> #################################### >> Verifying the location of autogen.sh... done >> Checking if autotools are in the same location... yes, all in /Users/gropp >> Checking for autoconf version... >= 2.67 >> Checking for automake version... >= 1.15 >> Checking for libtool version... >= 2.4.4 >> Checking for UNIX find... done >> Checking if xargs rm -rf works... yes >> ########################################################### >> ## Checking submodules >> ########################################################### >> ===> ERROR: Submodule src/hwloc is not checked out >> wdg-macbook:mpich-trunk gropp$ >> How exactly is src/hwloc to be checked out? There is nothing about that at https://wiki.mpich.org/mpich/index.php/Getting_And_Building_MPICH . Are up-to-date instructions available? >> Bill >> William Gropp >> Director and Chief Scientist, NCSA >> Thomas M. Siebel Chair in Computer Science >> University of Illinois Urbana-Champaign >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss

1 0