December 2017 - discuss - mailman3.cels.anl.gov

valgrind warning with ROMIO hint file
by Audet, Martin 19 Dec '17

19 Dec '17

MPI_Developers, It seems that when a program using mpich uses MPI-IO, it try first to open a hint file specified in the ROMIO_HINTS environment variable (or "/etc/romio-hints" if it is undefined) and then try to read it. The problem is that it will try to read from it with the file descriptor returned by open() even if the open() failed and returned a -1 (ex: file not found). When this happens read() also fails and also return -1. The problem is when the program is run under valgrind (with the valgrind friendly ch3:sock device), valgrind notice that an invalid file descriptor was passed to read() and complain with a warning: ==113473== Memcheck, a memory error detector ==113473== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==113473== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info ==113473== Command: ... ==113473== Parent PID: 48694 ==113473== ==113473== Warning: invalid file descriptor -1 in syscall read() ==113473== ==113473== HEAP SUMMARY: ==113473== in use at exit: 0 bytes in 0 blocks ==113473== total heap usage: 4,420 allocs, 4,209 frees, 72,241,561 bytes allocated ==113473== ==113473== All heap blocks were freed -- no leaks are possible ==113473== ==113473== For counts of detected and suppressed errors, rerun with: -v ==113473== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) The problem is that no matter how clean is the program, this warning will appear in its valgrind report. A developer may think that his own code is problematic and start investigating... The solution is of course not trying to read the file if the file descriptor is invalid. This could be done simply as follow (mpich 3.2.1): --- mpich-3.2.1/src/mpi/romio/adio/common/system_hints.c +++ mpich-3.2.1-patch/src/mpi/romio/adio/common/system_hints.c @@ -98,7 +98,7 @@ buffer = (char *)ADIOI_Calloc(HINTFILE_MAX_SIZE, sizeof (char)); if (rank == 0) { - ret = read(fd, buffer, HINTFILE_MAX_SIZE); + ret = (fd >= 0) ? read(fd, buffer, HINTFILE_MAX_SIZE) : -1; /* any error: bad/nonexistent fd, no perms, anything: set up a null * buffer and the subsequent string parsing will quit immediately */ if (ret == -1) Could you consider applying this patch (or an equivalent one) to mpich ? It would be much appreciated. Thanks in advance, Martin Audet _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] BUG with ROMIO on NFS since commit b4ab2f118d
by Jeff Hammond 14 Dec '17

14 Dec '17

Anyone with a GitHub account should be able to create an issue here: https://github.com/pmodels/mpich/issues/. Best, Jeff On Thu, Dec 14, 2017 at 8:21 AM, Eric Chamberland < Eric.Chamberland(a)giref.ulaval.ca> wrote: > Hi, > > Can I have an account to declare this bug into the bug tracker? > > Thanks, > > Eric > > > On 08/12/17 10:14 AM, Guo, Yanfei wrote: > >> Hi Eric, >> >> Sorry about the delay. I am able to reproduce the problem with your >> example (even with the master branch of MPICH). Maybe Rob can take a look >> at the problem. >> >> Yanfei Guo >> Assistant Computer Scientist >> MCS Division, ANL >> >> >> On 12/7/17, 1:59 PM, "Eric Chamberland" <Eric.Chamberland(a)giref.ulaval.ca> >> wrote: >> >> Hi, >> >> I first posted on Nov 15 this bug on the list and I still have no reply. >> >> Is there something I should know or is there a better place to post an >> MPICH bug? >> >> Thanks, >> >> Eric >> >> >> On 04/12/17 08:18 PM, Eric Chamberland wrote: >> >>> Hi, >>> >>> I have taken some time to have a relative "small" code to reproduce the >>> problem. >>> >>> The good thing is that now I can extract the exact call sequence of >>> almost any MPI I/O call we make in our code to have a pure "mpi" calls >>> in s simple C program, independent of our in-house code. >>> >>> To reproduce the bug, just compile the attached file with any >>> mpich/master since commit b4ab2f118d (nov 8) , and launch the resulting >>> executable with 3 processes along with the second attachment >>> (file_for_bug.data) saved in the pwd on an *NFS* path. >>> >>> You should see comething like this: >>> >>> ERROR Returned by MPI: 604040736 >>> ERROR_string Returned by MPI: Other I/O error , error stack: >>> ADIOI_NFS_READSTRIDED(523): Other I/O error Bad file descriptor >>> ERROR Returned by MPI: 268496416 >>> ERROR_string Returned by MPI: Other I/O error , error stack: >>> ADIOI_NFS_READSTRIDED(523): Other I/O error Operation now in progress >>> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >>> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 >>> >>> If you launch it on a local drive, it works. >>> >>> Can someone confirm it as reproduce the problem please? >>> >>> Moreover, if I launch it with valgrind, even on a local disk, it >>> complains like this, on process 0: >>> >>> ==99023== Warning: invalid file descriptor -1 in syscall read() >>> ==99023== at 0x53E2CB0: __read_nocancel (in /lib64/libc-2.19.so) >>> ==99023== by 0x5041606: file_to_info_all (system_hints.c:101) >>> ==99023== by 0x5041606: ADIOI_process_system_hints >>> (system_hints.c:150) >>> ==99023== by 0x50311B8: ADIO_Open (ad_open.c:123) >>> ==99023== by 0x50161FD: PMPI_File_open (open.c:154) >>> ==99023== by 0x400F91: main (mpich_mpiio_nfs_bug_read.c:42) >>> >>> Thanks, >>> >>> Eric >>> >>> On 21/11/17 02:49 PM, Eric Chamberland wrote: >>> >>>> Hi M. Latham, >>>> >>>> I have more information now. >>>> >>>> When I try to run my example on NFS, I have the following error code: >>>> >>>> error #812707360 >>>> Other I/O error , error stack: >>>> ADIOI_NFS_READSTRIDED(523): Other I/O error Success >>>> >>>> that is returned by MPI_File_read_all_begin >>>> >>>> When I try on a local disk, everything is fine. >>>> >>>> Here are all files about my actual build: >>>> >>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11. >>>> 21.05h40m02s_config.log >>>> >>>> >>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11. >>>> 21.05h40m02s_c.txt >>>> >>>> >>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11. >>>> 21.05h40m02s_m.txt >>>> >>>> >>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11. >>>> 21.05h40m02s_mi.txt >>>> >>>> >>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11. >>>> 21.05h40m02s_mpl_config.log >>>> >>>> >>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11. >>>> 21.05h40m02s_pm_hydra_config.log >>>> >>>> >>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11. >>>> 21.05h40m02s_mpiexec_info.txt >>>> >>>> >>>> Hope this help to dig further into this issue. >>>> >>>> Thanks, >>>> >>>> Eric >>>> >>>> >>>> On 15/11/17 03:55 PM, Eric Chamberland wrote: >>>> >>>>> Hi, >>>>> >>>>> We are compiling with mpich/master each night since august 2016... >>>>> >>>>> since nov 8, the mpich/master branch is buggy with our nighlty build >>>>> tests. >>>>> >>>>> Here is the nov 8 config.log: >>>>> >>>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11. >>>>> 08.05h36m02s_config.log >>>>> >>>>> >>>>> For nov 7 the configure log: >>>>> >>>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11. >>>>> 07.05h36m01s_config.log >>>>> >>>>> >>>>> >>>>> Since nov 8, on a specific ROMIO test, it is hanging indefinitely in >>>>> optimized mode, and into DEBUG mode, I have a strange (yet to be >>>>> debugged) assertion in our code. >>>>> >>>>> I reran the test manually, and when I wrote the results on a local >>>>> disk, everything is fine. >>>>> >>>>> However, when I write over *NFS*, the test is faulty. >>>>> >>>>> I have not yet debugged enough through this, but, I suspect something >>>>> related with one of: >>>>> >>>>> MPI_File_write_all_begin >>>>> MPI_File_write_all_end >>>>> MPI_File_read_all_begin >>>>> MPI_File_read_all_end >>>>> MPI_File_set_view >>>>> MPI_Type_free >>>>> >>>>> Am I alone to see these problems? >>>>> >>>>> Thanks, >>>>> Eric >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] BUG with ROMIO on NFS since commit b4ab2f118d
by Eric Chamberland 14 Dec '17

14 Dec '17

Hi, Can I have an account to declare this bug into the bug tracker? Thanks, Eric On 08/12/17 10:14 AM, Guo, Yanfei wrote: > Hi Eric, > > Sorry about the delay. I am able to reproduce the problem with your example (even with the master branch of MPICH). Maybe Rob can take a look at the problem. > > Yanfei Guo > Assistant Computer Scientist > MCS Division, ANL > > > > On 12/7/17, 1:59 PM, "Eric Chamberland" <Eric.Chamberland(a)giref.ulaval.ca> wrote: > > Hi, > > I first posted on Nov 15 this bug on the list and I still have no reply. > > Is there something I should know or is there a better place to post an > MPICH bug? > > Thanks, > > Eric > > > On 04/12/17 08:18 PM, Eric Chamberland wrote: >> Hi, >> >> I have taken some time to have a relative "small" code to reproduce the >> problem. >> >> The good thing is that now I can extract the exact call sequence of >> almost any MPI I/O call we make in our code to have a pure "mpi" calls >> in s simple C program, independent of our in-house code. >> >> To reproduce the bug, just compile the attached file with any >> mpich/master since commit b4ab2f118d (nov 8) , and launch the resulting >> executable with 3 processes along with the second attachment >> (file_for_bug.data) saved in the pwd on an *NFS* path. >> >> You should see comething like this: >> >> ERROR Returned by MPI: 604040736 >> ERROR_string Returned by MPI: Other I/O error , error stack: >> ADIOI_NFS_READSTRIDED(523): Other I/O error Bad file descriptor >> ERROR Returned by MPI: 268496416 >> ERROR_string Returned by MPI: Other I/O error , error stack: >> ADIOI_NFS_READSTRIDED(523): Other I/O error Operation now in progress >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 >> >> If you launch it on a local drive, it works. >> >> Can someone confirm it as reproduce the problem please? >> >> Moreover, if I launch it with valgrind, even on a local disk, it >> complains like this, on process 0: >> >> ==99023== Warning: invalid file descriptor -1 in syscall read() >> ==99023== at 0x53E2CB0: __read_nocancel (in /lib64/libc-2.19.so) >> ==99023== by 0x5041606: file_to_info_all (system_hints.c:101) >> ==99023== by 0x5041606: ADIOI_process_system_hints (system_hints.c:150) >> ==99023== by 0x50311B8: ADIO_Open (ad_open.c:123) >> ==99023== by 0x50161FD: PMPI_File_open (open.c:154) >> ==99023== by 0x400F91: main (mpich_mpiio_nfs_bug_read.c:42) >> >> Thanks, >> >> Eric >> >> On 21/11/17 02:49 PM, Eric Chamberland wrote: >>> Hi M. Latham, >>> >>> I have more information now. >>> >>> When I try to run my example on NFS, I have the following error code: >>> >>> error #812707360 >>> Other I/O error , error stack: >>> ADIOI_NFS_READSTRIDED(523): Other I/O error Success >>> >>> that is returned by MPI_File_read_all_begin >>> >>> When I try on a local disk, everything is fine. >>> >>> Here are all files about my actual build: >>> >>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_con… >>> >>> >>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_c.t… >>> >>> >>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_m.t… >>> >>> >>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mi.… >>> >>> >>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mpl… >>> >>> >>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_pm_… >>> >>> >>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mpi… >>> >>> >>> Hope this help to dig further into this issue. >>> >>> Thanks, >>> >>> Eric >>> >>> >>> On 15/11/17 03:55 PM, Eric Chamberland wrote: >>>> Hi, >>>> >>>> We are compiling with mpich/master each night since august 2016... >>>> >>>> since nov 8, the mpich/master branch is buggy with our nighlty build >>>> tests. >>>> >>>> Here is the nov 8 config.log: >>>> >>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.08.05h36m02s_con… >>>> >>>> >>>> For nov 7 the configure log: >>>> >>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.07.05h36m01s_con… >>>> >>>> >>>> >>>> Since nov 8, on a specific ROMIO test, it is hanging indefinitely in >>>> optimized mode, and into DEBUG mode, I have a strange (yet to be >>>> debugged) assertion in our code. >>>> >>>> I reran the test manually, and when I wrote the results on a local >>>> disk, everything is fine. >>>> >>>> However, when I write over *NFS*, the test is faulty. >>>> >>>> I have not yet debugged enough through this, but, I suspect something >>>> related with one of: >>>> >>>> MPI_File_write_all_begin >>>> MPI_File_write_all_end >>>> MPI_File_read_all_begin >>>> MPI_File_read_all_end >>>> MPI_File_set_view >>>> MPI_Type_free >>>> >>>> Am I alone to see these problems? >>>> >>>> Thanks, >>>> Eric >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Building MPICH with Flang
by Tarun Prabhu 09 Dec '17

09 Dec '17

I have been trying to build MPICH with Clang + Flang. If I build it with clang without Fortran support, it works fine, but if I try to build it with Flang for Fortran, I get an unknown argument error in the final link step. clang-5.0: error: unsupported option '--whole-archive' clang-5.0: error: unsupported option '--no-whole-archive' clang-5.0: error: unknown argument: '-soname' clang-5.0: error: no such file or directory: 'libmpi.so.12' This seems to be because --whole-archive is passed to clang without prepending it with -Wl,. So I hacked the libtool script to force it to add the "-Wl". Now I get another error saying: usr/bin/ld: src/binding/fortran/mpif_h/.libs/setbotf.o: relocation R_X86_64_32 against undefined symbol `mpipriv1_' can not be used when making a shared object; recompile with -fPIC Could this be because the build system does not recognize/support flang? If so, is there a way to work around it? If not, I'd appreciate some help in trying to get it built. Thanks, Tarun _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] how to run jobs using IPv6?
by Guo, Yanfei 08 Dec '17

08 Dec '17

Hi Shuwei, Thanks for the information. Yanfei Guo Assistant Computer Scientist MCS Division, ANL On 12/8/17, 3:17 PM, "Shuwei Zhao" <shuweizhao1991(a)gmail.com> wrote: Hi Yanfei， Just friendly reminder If mpich contain the functions below which only supports IPv4, please replace them together gethostbyname/getservbyname => getaddrinfo gethostbyaddr => getnameinfo inet_ntoa => inet_ntop inet_aton => inet_pton socket/bind/accept/recv/recvfrom/send/sendto/ => API argument list changed I also comment under the ticket. Thanks Shuwei On Wed, 6 Dec 2017 at 12:21 Shuwei Zhao <shuweizhao1991(a)gmail.com> wrote: Hi yanfei Thanks for creating the ticket. Do you have any idea on how long this ticket will be handled and closed? I think for this ticket the code change should not be big. This feature is pretty urgent for my team. Thanks On Wed, 6 Dec 2017 at 12:07 Guo, Yanfei <yguo(a)anl.gov> wrote: I have created an ticket to track it. https://github.com/pmodels/mpich/issues/2889 <https://github.com/pmodels/mpich/issues/2889> Yanfei Guo Assistant Computer Scientist MCS Division, ANL On 12/6/17, 11:52 AM, "Shuwei Zhao" <shuweizhao1991(a)gmail.com> wrote: Are you going to use getaddrinfo instead to deprecate gethostbyname? Thanks On Wed, 6 Dec 2017 at 11:50 Guo, Yanfei <yguo(a)anl.gov> wrote: Hi Shuwei, The communication over IPv6 network is supported as long as the lower-level network library supports it. However, MPICH uses gethostbyname to translate hostnames to addresses in order to initialize the link which is not working on pure IPv6. Yanfei Guo Assistant Computer Scientist MCS Division, ANL On 12/6/17, 11:46 AM, "Shuwei Zhao" <shuweizhao1991(a)gmail.com> wrote: Hi Yanfei， I ask this question months ago，but I was told IPv6 is supported in mpich 3.*. So it's not supported? Thanks Shuwei On Wed, 6 Dec 2017 at 11:24 Guo, Yanfei <yguo(a)anl.gov> wrote: Hi Carlos, MPICH currently uses gethostbyname which does not support IPv6, therefore it won’t work on a pure IPv6 network. Best, Yanfei Guo Assistant Computer Scientist MCS Division, ANL On 11/22/17, 2:23 PM, "Carlos" <carlos(a)inf.ufpr.br> wrote: Is it possible to use mpich (3.3) with IPv6-only? We tried it here and got Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(506)..............: MPID_Init(211).....................: channel initialization failed MPIDI_CH3_Init(89).................: MPID_nem_init(324).................: MPID_nem_tcp_init(178).............: MPID_nem_tcp_get_business_card(425): ckpt_restart(384)..................: gethostbyname failed, zuratkul (errno 4) ************* This shows that mpich doesn't run without IPv4, which is our case... Is there a way to make it work in this situation? _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] how to run jobs using IPv6?
by Shuwei Zhao 08 Dec '17

08 Dec '17

Hi Yanfei， Just friendly reminder If mpich contain the functions below which only supports IPv4, please replace them together gethostbyname/getservbyname => getaddrinfo gethostbyaddr => getnameinfo inet_ntoa => inet_ntop inet_aton => inet_pton socket/bind/accept/recv/recvfrom/send/sendto/ => API argument list changed I also comment under the ticket. Thanks Shuwei On Wed, 6 Dec 2017 at 12:21 Shuwei Zhao <shuweizhao1991(a)gmail.com> wrote: > Hi yanfei > Thanks for creating the ticket. > Do you have any idea on how long this ticket will be handled and closed? > I think for this ticket the code change should not be big. This feature is > pretty urgent for my team. > Thanks > > On Wed, 6 Dec 2017 at 12:07 Guo, Yanfei <yguo(a)anl.gov> wrote: > >> I have created an ticket to track it. >> https://github.com/pmodels/mpich/issues/2889 >> >> Yanfei Guo >> Assistant Computer Scientist >> MCS Division, ANL >> >> >> >> On 12/6/17, 11:52 AM, "Shuwei Zhao" <shuweizhao1991(a)gmail.com> wrote: >> >> Are you going to use getaddrinfo instead to deprecate gethostbyname? >> >> >> Thanks >> >> On Wed, 6 Dec 2017 at 11:50 Guo, Yanfei <yguo(a)anl.gov> wrote: >> >> >> Hi Shuwei, >> >> The communication over IPv6 network is supported as long as the >> lower-level network library supports it. However, MPICH uses gethostbyname >> to translate hostnames to addresses in order to initialize the link which >> is not working on pure IPv6. >> >> Yanfei Guo >> Assistant Computer Scientist >> MCS Division, ANL >> >> >> >> On 12/6/17, 11:46 AM, "Shuwei Zhao" <shuweizhao1991(a)gmail.com> wrote: >> >> Hi Yanfei， >> >> >> I ask this question months ago，but I was told IPv6 is supported in mpich >> 3.*. So it's not supported? >> >> >> Thanks >> Shuwei >> >> On Wed, 6 Dec 2017 at 11:24 Guo, Yanfei <yguo(a)anl.gov> wrote: >> >> >> Hi Carlos, >> >> MPICH currently uses gethostbyname which does not support IPv6, therefore >> it won’t work on a pure IPv6 network. >> >> Best, >> >> Yanfei Guo >> Assistant Computer Scientist >> MCS Division, ANL >> >> >> >> On 11/22/17, 2:23 PM, "Carlos" <carlos(a)inf.ufpr.br> wrote: >> >> Is it possible to use mpich (3.3) with IPv6-only? We tried it here and got >> >> Fatal error in MPI_Init: Other MPI error, error stack: >> MPIR_Init_thread(506)..............: >> MPID_Init(211).....................: channel initialization failed >> MPIDI_CH3_Init(89).................: >> MPID_nem_init(324).................: >> MPID_nem_tcp_init(178).............: >> MPID_nem_tcp_get_business_card(425): >> ckpt_restart(384)..................: gethostbyname failed, zuratkul >> (errno 4) >> ************* >> >> This shows that mpich doesn't run without IPv4, which is our case... Is >> there a >> way to make it work in this situation? >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] BUG with ROMIO on NFS since commit b4ab2f118d
by Guo, Yanfei 08 Dec '17

08 Dec '17

Hi Eric, Sorry about the delay. I am able to reproduce the problem with your example (even with the master branch of MPICH). Maybe Rob can take a look at the problem. Yanfei Guo Assistant Computer Scientist MCS Division, ANL On 12/7/17, 1:59 PM, "Eric Chamberland" <Eric.Chamberland(a)giref.ulaval.ca> wrote: Hi, I first posted on Nov 15 this bug on the list and I still have no reply. Is there something I should know or is there a better place to post an MPICH bug? Thanks, Eric On 04/12/17 08:18 PM, Eric Chamberland wrote: > Hi, > > I have taken some time to have a relative "small" code to reproduce the > problem. > > The good thing is that now I can extract the exact call sequence of > almost any MPI I/O call we make in our code to have a pure "mpi" calls > in s simple C program, independent of our in-house code. > > To reproduce the bug, just compile the attached file with any > mpich/master since commit b4ab2f118d (nov 8) , and launch the resulting > executable with 3 processes along with the second attachment > (file_for_bug.data) saved in the pwd on an *NFS* path. > > You should see comething like this: > > ERROR Returned by MPI: 604040736 > ERROR_string Returned by MPI: Other I/O error , error stack: > ADIOI_NFS_READSTRIDED(523): Other I/O error Bad file descriptor > ERROR Returned by MPI: 268496416 > ERROR_string Returned by MPI: Other I/O error , error stack: > ADIOI_NFS_READSTRIDED(523): Other I/O error Operation now in progress > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 > > If you launch it on a local drive, it works. > > Can someone confirm it as reproduce the problem please? > > Moreover, if I launch it with valgrind, even on a local disk, it > complains like this, on process 0: > > ==99023== Warning: invalid file descriptor -1 in syscall read() > ==99023== at 0x53E2CB0: __read_nocancel (in /lib64/libc-2.19.so) > ==99023== by 0x5041606: file_to_info_all (system_hints.c:101) > ==99023== by 0x5041606: ADIOI_process_system_hints (system_hints.c:150) > ==99023== by 0x50311B8: ADIO_Open (ad_open.c:123) > ==99023== by 0x50161FD: PMPI_File_open (open.c:154) > ==99023== by 0x400F91: main (mpich_mpiio_nfs_bug_read.c:42) > > Thanks, > > Eric > > On 21/11/17 02:49 PM, Eric Chamberland wrote: >> Hi M. Latham, >> >> I have more information now. >> >> When I try to run my example on NFS, I have the following error code: >> >> error #812707360 >> Other I/O error , error stack: >> ADIOI_NFS_READSTRIDED(523): Other I/O error Success >> >> that is returned by MPI_File_read_all_begin >> >> When I try on a local disk, everything is fine. >> >> Here are all files about my actual build: >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_con… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_c.t… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_m.t… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mi.… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mpl… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_pm_… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mpi… >> >> >> Hope this help to dig further into this issue. >> >> Thanks, >> >> Eric >> >> >> On 15/11/17 03:55 PM, Eric Chamberland wrote: >>> Hi, >>> >>> We are compiling with mpich/master each night since august 2016... >>> >>> since nov 8, the mpich/master branch is buggy with our nighlty build >>> tests. >>> >>> Here is the nov 8 config.log: >>> >>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.08.05h36m02s_con… >>> >>> >>> For nov 7 the configure log: >>> >>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.07.05h36m01s_con… >>> >>> >>> >>> Since nov 8, on a specific ROMIO test, it is hanging indefinitely in >>> optimized mode, and into DEBUG mode, I have a strange (yet to be >>> debugged) assertion in our code. >>> >>> I reran the test manually, and when I wrote the results on a local >>> disk, everything is fine. >>> >>> However, when I write over *NFS*, the test is faulty. >>> >>> I have not yet debugged enough through this, but, I suspect something >>> related with one of: >>> >>> MPI_File_write_all_begin >>> MPI_File_write_all_end >>> MPI_File_read_all_begin >>> MPI_File_read_all_end >>> MPI_File_set_view >>> MPI_Type_free >>> >>> Am I alone to see these problems? >>> >>> Thanks, >>> Eric >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] BUG with ROMIO on NFS since commit b4ab2f118d
by Eric Chamberland 07 Dec '17

07 Dec '17

Hi, I first posted on Nov 15 this bug on the list and I still have no reply. Is there something I should know or is there a better place to post an MPICH bug? Thanks, Eric On 04/12/17 08:18 PM, Eric Chamberland wrote: > Hi, > > I have taken some time to have a relative "small" code to reproduce the > problem. > > The good thing is that now I can extract the exact call sequence of > almost any MPI I/O call we make in our code to have a pure "mpi" calls > in s simple C program, independent of our in-house code. > > To reproduce the bug, just compile the attached file with any > mpich/master since commit b4ab2f118d (nov 8) , and launch the resulting > executable with 3 processes along with the second attachment > (file_for_bug.data) saved in the pwd on an *NFS* path. > > You should see comething like this: > > ERROR Returned by MPI: 604040736 > ERROR_string Returned by MPI: Other I/O error , error stack: > ADIOI_NFS_READSTRIDED(523): Other I/O error Bad file descriptor > ERROR Returned by MPI: 268496416 > ERROR_string Returned by MPI: Other I/O error , error stack: > ADIOI_NFS_READSTRIDED(523): Other I/O error Operation now in progress > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 > > If you launch it on a local drive, it works. > > Can someone confirm it as reproduce the problem please? > > Moreover, if I launch it with valgrind, even on a local disk, it > complains like this, on process 0: > > ==99023== Warning: invalid file descriptor -1 in syscall read() > ==99023== at 0x53E2CB0: __read_nocancel (in /lib64/libc-2.19.so) > ==99023== by 0x5041606: file_to_info_all (system_hints.c:101) > ==99023== by 0x5041606: ADIOI_process_system_hints (system_hints.c:150) > ==99023== by 0x50311B8: ADIO_Open (ad_open.c:123) > ==99023== by 0x50161FD: PMPI_File_open (open.c:154) > ==99023== by 0x400F91: main (mpich_mpiio_nfs_bug_read.c:42) > > Thanks, > > Eric > > On 21/11/17 02:49 PM, Eric Chamberland wrote: >> Hi M. Latham, >> >> I have more information now. >> >> When I try to run my example on NFS, I have the following error code: >> >> error #812707360 >> Other I/O error , error stack: >> ADIOI_NFS_READSTRIDED(523): Other I/O error Success >> >> that is returned by MPI_File_read_all_begin >> >> When I try on a local disk, everything is fine. >> >> Here are all files about my actual build: >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_con… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_c.t… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_m.t… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mi.… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mpl… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_pm_… >> >> >> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mpi… >> >> >> Hope this help to dig further into this issue. >> >> Thanks, >> >> Eric >> >> >> On 15/11/17 03:55 PM, Eric Chamberland wrote: >>> Hi, >>> >>> We are compiling with mpich/master each night since august 2016... >>> >>> since nov 8, the mpich/master branch is buggy with our nighlty build >>> tests. >>> >>> Here is the nov 8 config.log: >>> >>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.08.05h36m02s_con… >>> >>> >>> For nov 7 the configure log: >>> >>> http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.07.05h36m01s_con… >>> >>> >>> >>> Since nov 8, on a specific ROMIO test, it is hanging indefinitely in >>> optimized mode, and into DEBUG mode, I have a strange (yet to be >>> debugged) assertion in our code. >>> >>> I reran the test manually, and when I wrote the results on a local >>> disk, everything is fine. >>> >>> However, when I write over *NFS*, the test is faulty. >>> >>> I have not yet debugged enough through this, but, I suspect something >>> related with one of: >>> >>> MPI_File_write_all_begin >>> MPI_File_write_all_end >>> MPI_File_read_all_begin >>> MPI_File_read_all_end >>> MPI_File_set_view >>> MPI_Type_free >>> >>> Am I alone to see these problems? >>> >>> Thanks, >>> Eric >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] how to run jobs using IPv6?
by Carlos 06 Dec '17

06 Dec '17

Shuwei Zhao (shuweizhao1991(a)gmail.com) wrote on Wed, Dec 06, 2017 at 04:21:42PM BRST: > Thanks for creating the ticket. > Do you have any idea on how long this ticket will be handled and closed? > I think for this ticket the code change should not be big. This feature is > pretty urgent for my team. This is crucial for us too. We cannot use mpich because of this. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] how to run jobs using IPv6?
by Shuwei Zhao 06 Dec '17

06 Dec '17

Hi yanfei Thanks for creating the ticket. Do you have any idea on how long this ticket will be handled and closed? I think for this ticket the code change should not be big. This feature is pretty urgent for my team. Thanks On Wed, 6 Dec 2017 at 12:07 Guo, Yanfei <yguo(a)anl.gov> wrote: > I have created an ticket to track it. > https://github.com/pmodels/mpich/issues/2889 > > Yanfei Guo > Assistant Computer Scientist > MCS Division, ANL > > > > On 12/6/17, 11:52 AM, "Shuwei Zhao" <shuweizhao1991(a)gmail.com> wrote: > > Are you going to use getaddrinfo instead to deprecate gethostbyname? > > > Thanks > > On Wed, 6 Dec 2017 at 11:50 Guo, Yanfei <yguo(a)anl.gov> wrote: > > > Hi Shuwei, > > The communication over IPv6 network is supported as long as the > lower-level network library supports it. However, MPICH uses gethostbyname > to translate hostnames to addresses in order to initialize the link which > is not working on pure IPv6. > > Yanfei Guo > Assistant Computer Scientist > MCS Division, ANL > > > > On 12/6/17, 11:46 AM, "Shuwei Zhao" <shuweizhao1991(a)gmail.com> wrote: > > Hi Yanfei， > > > I ask this question months ago，but I was told IPv6 is supported in mpich > 3.*. So it's not supported? > > > Thanks > Shuwei > > On Wed, 6 Dec 2017 at 11:24 Guo, Yanfei <yguo(a)anl.gov> wrote: > > > Hi Carlos, > > MPICH currently uses gethostbyname which does not support IPv6, therefore > it won’t work on a pure IPv6 network. > > Best, > > Yanfei Guo > Assistant Computer Scientist > MCS Division, ANL > > > > On 11/22/17, 2:23 PM, "Carlos" <carlos(a)inf.ufpr.br> wrote: > > Is it possible to use mpich (3.3) with IPv6-only? We tried it here and got > > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(506)..............: > MPID_Init(211).....................: channel initialization failed > MPIDI_CH3_Init(89).................: > MPID_nem_init(324).................: > MPID_nem_tcp_init(178).............: > MPID_nem_tcp_get_business_card(425): > ckpt_restart(384)..................: gethostbyname failed, zuratkul (errno > 4) > ************* > > This shows that mpich doesn't run without IPv4, which is our case... Is > there a > way to make it work in this situation? > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0