December 2017 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] valgrind warning with ROMIO hint file
by Audet, Martin 22 Dec '17

22 Dec '17

Hi Eric, At the beginning I thought that your issue was related but different. But now like you I think it is the same. When I ran your mpich_mpiio_file_descriptor_bug.c program with valgrind compiled with the unmodified mpich 3.2.1 (ch3:sock device), I get the same warning message about read() using an invalid file descriptor (-1): $ mpicc mpich_mpiio_file_descriptor_bug.c $ valgrind ./a.out ==118187== Memcheck, a memory error detector ==118187== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==118187== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info ==118187== Command: ./a.out ==118187== ==118187== Warning: invalid file descriptor -1 in syscall read() ==118187== ==118187== HEAP SUMMARY: ==118187== in use at exit: 0 bytes in 0 blocks ==118187== total heap usage: 558 allocs, 527 frees, 21,101,877 bytes allocated ==118187== ==118187== All heap blocks were freed -- no leaks are possible ==118187== ==118187== For counts of detected and suppressed errors, rerun with: -v ==118187== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) However when I compile your program with a modified version of mpich 3.2.1 (same ch3:sock device) to avoid trying to read the romio configuration file if the file descriptor is invalid (the one line patch I shown in my original message), the valgrind warning about invalid file descriptor with read() disappear. If we use the strace command to investigate what is happening (this command shows the system calls of another command and its return code) we see that with the unmodified mpich 3.2.1 (at the end of the long system call list): $ mpicc mpich_mpiio_file_descriptor_bug.c $ strace ./a.out open("/etc/romio-hints", O_RDONLY) = -1 ENOENT (No such file or directory) read(-1, 0x1ae2890, 4096) = -1 EBADF (Bad file descriptor) brk(NULL) = 0x1ae7000 brk(0x1b14000) = 0x1b14000 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5f60e6a000 umask(022) = 027 umask(027) = 022 open("touch.txt", O_RDONLY) = 4 close(4) = 0 munmap(0x7f5f60e6a000, 16781312) = 0 fcntl(3, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK) fcntl(3, F_SETFL, O_RDWR) = 0 close(3) = 0 munmap(0x7f5f61e6b000, 2756608) = 0 munmap(0x7f5f630b4000, 1314816) = 0 exit_group(0) = ? We see that the problematic -1 file descriptor is passed to read() for reading the ROMIO configuration file "/etc/romio-hints" (which doesn't exist on my machine), not to a system call associated with your "touch.txt" file. If instead we compile with my patched version of mpich 3.2.1, the problematic read() is gone: $ mpicc mpich_mpiio_file_descriptor_bug.c $ strace ./a.out open("/etc/romio-hints", O_RDONLY) = -1 ENOENT (No such file or directory) brk(NULL) = 0x1297000 brk(0x12c4000) = 0x12c4000 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f16ddffb000 umask(022) = 027 umask(027) = 022 open("touch.txt", O_RDONLY) = 4 close(4) = 0 munmap(0x7f16ddffb000, 16781312) = 0 fcntl(3, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK) fcntl(3, F_SETFL, O_RDWR) = 0 close(3) = 0 munmap(0x7f16deffc000, 2756608) = 0 munmap(0x7f16e0245000, 1314816) = 0 exit_group(0) = ? +++ exited with 0 +++ And of course valgrind doesn't complain anymore: $ valgrind ./a.out ==117748== Memcheck, a memory error detector ==117748== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==117748== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info ==117748== Command: ./a.out ==117748== ==117748== ==117748== HEAP SUMMARY: ==117748== in use at exit: 0 bytes in 0 blocks ==117748== total heap usage: 558 allocs, 527 frees, 21,101,877 bytes allocated ==117748== ==117748== All heap blocks were freed -- no leaks are possible ==117748== ==117748== For counts of detected and suppressed errors, rerun with: -v ==117748== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) I think the MPICH_Developers should apply my patch (or something equivalent). Joyeuses fêtes... Martin Audet -----Original Message----- From: Eric Chamberland [mailto:[email protected]] Sent: December 21, 2017 14:36 To: discuss(a)mpich.org Cc: Audet, Martin Subject: Re: [mpich-discuss] valgrind warning with ROMIO hint file Hi, we have the same issue here... Reported this 3 days ago: https://github.com/pmodels/mpich/issues/2894 Eric On 19/12/17 07:30 PM, Audet, Martin wrote: > MPI_Developers, > It seems that when a program using mpich uses MPI-IO, it try first to > open a hint file specified in the ROMIO_HINTS environment variable (or > "/etc/romio-hints" if it is undefined) and then try to read it. The > problem is that it will try to read from it with the file descriptor > returned by > open() even if the open() failed and returned a -1 (ex: file not found). > When this happens read() also fails and also return -1. > The problem is when the program is run under valgrind (with the > valgrind friendly ch3:sock device), valgrind notice that an invalid > file descriptor was passed to read() and complain with a warning: > ==113473== Memcheck, a memory error detector ==113473== Copyright (C) > 2002-2015, and GNU GPL'd, by Julian Seward et al. > ==113473== Using Valgrind-3.12.0 and LibVEX; rerun with -h for > copyright info ==113473== Command: ... > ==113473== Parent PID: 48694 > ==113473== > ==113473== Warning: invalid file descriptor -1 in syscall read() > ==113473== ==113473== HEAP SUMMARY: > ==113473== in use at exit: 0 bytes in 0 blocks > ==113473== total heap usage: 4,420 allocs, 4,209 frees, 72,241,561 > bytes allocated > ==113473== > ==113473== All heap blocks were freed -- no leaks are possible > ==113473== ==113473== For counts of detected and suppressed errors, > rerun with: -v ==113473== ERROR SUMMARY: 0 errors from 0 contexts > (suppressed: 0 from 0) The problem is that no matter how clean is the > program, this warning will appear in its valgrind report. A developer > may think that his own code is problematic and start investigating... > The solution is of course not trying to read the file if the file > descriptor is invalid. This could be done simply as follow (mpich 3.2.1): > --- mpich-3.2.1/src/mpi/romio/adio/common/system_hints.c > +++ mpich-3.2.1-patch/src/mpi/romio/adio/common/system_hints.c > @@ -98,7 +98,7 @@ > buffer = (char *)ADIOI_Calloc(HINTFILE_MAX_SIZE, sizeof (char)); > if (rank == 0) { > - ret = read(fd, buffer, HINTFILE_MAX_SIZE); > + ret = (fd >= 0) ? read(fd, buffer, HINTFILE_MAX_SIZE) : -1; > /* any error: bad/nonexistent fd, no perms, anything: set up a null > * buffer and the subsequent string parsing will quit > immediately */ > if (ret == -1) > Could you consider applying this patch (or an equivalent one) to mpich > ? It would be much appreciated. > Thanks in advance, > Martin Audet _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] valgrind warning with ROMIO hint file
by 123 21 Dec '17

21 Dec '17

Hi Eric, Thank you for your reply. But it seems that that is not the same as my issue... I'm even unable to 'make' the source code. Probably there's something wrong with makefile files? Sheng At 2017-12-22 03:36:00, "Eric Chamberland" <Eric.Chamberland(a)giref.ulaval.ca> wrote: >Hi, > >we have the same issue here... > >Reported this 3 days ago: > >https://github.com/pmodels/mpich/issues/2894 > >Eric > >On 19/12/17 07:30 PM, Audet, Martin wrote: >> MPI_Developers, >> It seems that when a program using mpich uses MPI-IO, it try first to open >> a hint file specified in the ROMIO_HINTS environment variable (or >> "/etc/romio-hints" if it is undefined) and then try to read it. The problem >> is that it will try to read from it with the file descriptor returned by >> open() even if the open() failed and returned a -1 (ex: file not found). >> When this happens read() also fails and also return -1. >> The problem is when the program is run under valgrind (with the valgrind >> friendly ch3:sock device), valgrind notice that an invalid file descriptor >> was passed to read() and complain with a warning: >> ==113473== Memcheck, a memory error detector >> ==113473== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. >> ==113473== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright >> info >> ==113473== Command: ... >> ==113473== Parent PID: 48694 >> ==113473== >> ==113473== Warning: invalid file descriptor -1 in syscall read() >> ==113473== >> ==113473== HEAP SUMMARY: >> ==113473== in use at exit: 0 bytes in 0 blocks >> ==113473== total heap usage: 4,420 allocs, 4,209 frees, 72,241,561 >> bytes allocated >> ==113473== >> ==113473== All heap blocks were freed -- no leaks are possible >> ==113473== >> ==113473== For counts of detected and suppressed errors, rerun with: -v >> ==113473== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) >> The problem is that no matter how clean is the program, this warning will >> appear in its valgrind report. A developer may think that his own code is >> problematic and start investigating... >> The solution is of course not trying to read the file if the file descriptor >> is invalid. This could be done simply as follow (mpich 3.2.1): >> --- mpich-3.2.1/src/mpi/romio/adio/common/system_hints.c >> +++ mpich-3.2.1-patch/src/mpi/romio/adio/common/system_hints.c >> @@ -98,7 +98,7 @@ >> buffer = (char *)ADIOI_Calloc(HINTFILE_MAX_SIZE, sizeof (char)); >> if (rank == 0) { >> - ret = read(fd, buffer, HINTFILE_MAX_SIZE); >> + ret = (fd >= 0) ? read(fd, buffer, HINTFILE_MAX_SIZE) : -1; >> /* any error: bad/nonexistent fd, no perms, anything: set up a null >> * buffer and the subsequent string parsing will quit >> immediately */ >> if (ret == -1) >> Could you consider applying this patch (or an equivalent one) >> to mpich ? It would be much appreciated. >> Thanks in advance, >> Martin Audet >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error reporting
by 123 21 Dec '17

21 Dec '17

Hi Guo, Sorry I forgot to tell you about this. I'm trying it on CentOS 6.9 x86_64, and the filesystem is ext4. I'm learing about it so I am using a virtual machine. Will that matter? Sheng At 2017-12-22 07:14:16, "Guo, Yanfei" <yguo(a)anl.gov> wrote: >Hi Sheng, > >What platform were you trying this on (OS, filesystem)? > >Yanfei Guo >Assistant Computer Scientist >MCS Division, ANL > > > >On 12/21/17, 6:14 AM, "123" <a5467021(a)163.com> wrote: > >Hello developers of MPICH, > >I am a student who came to know about MPI shortly before, and I'm trying to install >MPICH as my MPI environment. The MPICH version is 3.2.1. But when I'm installing >the package following your documents, I encountered a problem when running make >command. >The files required are attached to this e-mail, and it would be very nice to have your >early reply. > >Regards, >Sheng > > > > > > > > > > > >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error reporting
by 123 21 Dec '17

21 Dec '17

Hi Guo, Sorry I forgot to tell you about this. I'm trying it on CentOS 6.9 x86_64, and the filesystem is ext4. I'm learing about it so I am using a virtual machine. Will that matter? Sheng At 2017-12-22 07:14:16, "Guo, Yanfei" <yguo(a)anl.gov> wrote: >Hi Sheng, > >What platform were you trying this on (OS, filesystem)? > >Yanfei Guo >Assistant Computer Scientist >MCS Division, ANL > > > >On 12/21/17, 6:14 AM, "123" <a5467021(a)163.com> wrote: > >Hello developers of MPICH, > >I am a student who came to know about MPI shortly before, and I'm trying to install >MPICH as my MPI environment. The MPICH version is 3.2.1. But when I'm installing >the package following your documents, I encountered a problem when running make >command. >The files required are attached to this e-mail, and it would be very nice to have your >early reply. > >Regards, >Sheng > > > > > > > > > > > >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] valgrind warning with ROMIO hint file
by 123 21 Dec '17

21 Dec '17

Hi Eric, Thank you for your reply. But it seems that that is not the same as my issue... I'm even unable to 'make' the source code. Probably there's something wrong with makefile files? Sheng At 2017-12-22 03:36:00, "Eric Chamberland" <Eric.Chamberland(a)giref.ulaval.ca> wrote: >Hi, > >we have the same issue here... > >Reported this 3 days ago: > >https://github.com/pmodels/mpich/issues/2894 > >Eric > >On 19/12/17 07:30 PM, Audet, Martin wrote: >> MPI_Developers, >> It seems that when a program using mpich uses MPI-IO, it try first to open >> a hint file specified in the ROMIO_HINTS environment variable (or >> "/etc/romio-hints" if it is undefined) and then try to read it. The problem >> is that it will try to read from it with the file descriptor returned by >> open() even if the open() failed and returned a -1 (ex: file not found). >> When this happens read() also fails and also return -1. >> The problem is when the program is run under valgrind (with the valgrind >> friendly ch3:sock device), valgrind notice that an invalid file descriptor >> was passed to read() and complain with a warning: >> ==113473== Memcheck, a memory error detector >> ==113473== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. >> ==113473== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright >> info >> ==113473== Command: ... >> ==113473== Parent PID: 48694 >> ==113473== >> ==113473== Warning: invalid file descriptor -1 in syscall read() >> ==113473== >> ==113473== HEAP SUMMARY: >> ==113473== in use at exit: 0 bytes in 0 blocks >> ==113473== total heap usage: 4,420 allocs, 4,209 frees, 72,241,561 >> bytes allocated >> ==113473== >> ==113473== All heap blocks were freed -- no leaks are possible >> ==113473== >> ==113473== For counts of detected and suppressed errors, rerun with: -v >> ==113473== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) >> The problem is that no matter how clean is the program, this warning will >> appear in its valgrind report. A developer may think that his own code is >> problematic and start investigating... >> The solution is of course not trying to read the file if the file descriptor >> is invalid. This could be done simply as follow (mpich 3.2.1): >> --- mpich-3.2.1/src/mpi/romio/adio/common/system_hints.c >> +++ mpich-3.2.1-patch/src/mpi/romio/adio/common/system_hints.c >> @@ -98,7 +98,7 @@ >> buffer = (char *)ADIOI_Calloc(HINTFILE_MAX_SIZE, sizeof (char)); >> if (rank == 0) { >> - ret = read(fd, buffer, HINTFILE_MAX_SIZE); >> + ret = (fd >= 0) ? read(fd, buffer, HINTFILE_MAX_SIZE) : -1; >> /* any error: bad/nonexistent fd, no perms, anything: set up a null >> * buffer and the subsequent string parsing will quit >> immediately */ >> if (ret == -1) >> Could you consider applying this patch (or an equivalent one) >> to mpich ? It would be much appreciated. >> Thanks in advance, >> Martin Audet >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error reporting
by Guo, Yanfei 21 Dec '17

21 Dec '17

Hi Sheng, What platform were you trying this on (OS, filesystem)? Yanfei Guo Assistant Computer Scientist MCS Division, ANL On 12/21/17, 6:14 AM, "123" <a5467021(a)163.com> wrote: Hello developers of MPICH, I am a student who came to know about MPI shortly before, and I'm trying to install MPICH as my MPI environment. The MPICH version is 3.2.1. But when I'm installing the package following your documents, I encountered a problem when running make command. The files required are attached to this e-mail, and it would be very nice to have your early reply. Regards, Sheng _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] valgrind warning with ROMIO hint file
by Eric Chamberland 21 Dec '17

21 Dec '17

Hi, we have the same issue here... Reported this 3 days ago: https://github.com/pmodels/mpich/issues/2894 Eric On 19/12/17 07:30 PM, Audet, Martin wrote: > MPI_Developers, > It seems that when a program using mpich uses MPI-IO, it try first to open > a hint file specified in the ROMIO_HINTS environment variable (or > "/etc/romio-hints" if it is undefined) and then try to read it. The problem > is that it will try to read from it with the file descriptor returned by > open() even if the open() failed and returned a -1 (ex: file not found). > When this happens read() also fails and also return -1. > The problem is when the program is run under valgrind (with the valgrind > friendly ch3:sock device), valgrind notice that an invalid file descriptor > was passed to read() and complain with a warning: > ==113473== Memcheck, a memory error detector > ==113473== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. > ==113473== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright > info > ==113473== Command: ... > ==113473== Parent PID: 48694 > ==113473== > ==113473== Warning: invalid file descriptor -1 in syscall read() > ==113473== > ==113473== HEAP SUMMARY: > ==113473== in use at exit: 0 bytes in 0 blocks > ==113473== total heap usage: 4,420 allocs, 4,209 frees, 72,241,561 > bytes allocated > ==113473== > ==113473== All heap blocks were freed -- no leaks are possible > ==113473== > ==113473== For counts of detected and suppressed errors, rerun with: -v > ==113473== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) > The problem is that no matter how clean is the program, this warning will > appear in its valgrind report. A developer may think that his own code is > problematic and start investigating... > The solution is of course not trying to read the file if the file descriptor > is invalid. This could be done simply as follow (mpich 3.2.1): > --- mpich-3.2.1/src/mpi/romio/adio/common/system_hints.c > +++ mpich-3.2.1-patch/src/mpi/romio/adio/common/system_hints.c > @@ -98,7 +98,7 @@ > buffer = (char *)ADIOI_Calloc(HINTFILE_MAX_SIZE, sizeof (char)); > if (rank == 0) { > - ret = read(fd, buffer, HINTFILE_MAX_SIZE); > + ret = (fd >= 0) ? read(fd, buffer, HINTFILE_MAX_SIZE) : -1; > /* any error: bad/nonexistent fd, no perms, anything: set up a null > * buffer and the subsequent string parsing will quit > immediately */ > if (ret == -1) > Could you consider applying this patch (or an equivalent one) > to mpich ? It would be much appreciated. > Thanks in advance, > Martin Audet _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Install of mpich-3.2.1 on ARM
by Kevin Olson 21 Dec '17

21 Dec '17

Hello, I’m attempting to install mpich-3.2.1 on an ARM system using the armclang compiler. The configure works OK, but I get the following error during the build: GEN lib/libmpi.la clang-5.0: error: unsupported option '--whole-archive' clang-5.0: error: unsupported option '--no-whole-archive' clang-5.0: error: unknown argument: '-soname' clang-5.0: error: no such file or directory: 'libmpi.so.12' Makefile:16369: recipe for target 'lib/libmpi.la' failed make[2]: *** [lib/libmpi.la] Error 1 make[2]: Leaving directory '/fserver/kevino/mpich-3.2.1' Makefile:38608: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/fserver/kevino/mpich-3.2.1' Makefile:10337: recipe for target 'all' failed make: *** [all] Error 2 Any suggestions? Kevin Olson Technical Consultant Numerical Algorithms Group, Inc 801 Warrenville Road, Suite 185 Lisle, Illinois 60532-4332 T 267-664-9913 F 630-971-2706 www.nag.com This e-mail has been scanned for all viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Error reporting
by 123 21 Dec '17

21 Dec '17

Hello developers of MPICH, I am a student who came to know about MPI shortly before, and I'm trying to install MPICH as my MPI environment. The MPICH version is 3.2.1. But when I'm installing the package following your documents, I encountered a problem when running make command. The files required are attached to this e-mail, and it would be very nice to have your early reply. Regards, Sheng _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_File_read_shared on NFS seems to be slow on multi-node
by Aboorva Devarajan 20 Dec '17

20 Dec '17

MPI_File_read_shared / MPI_File_write_shared on NFS seems to be very slow when running on multi-node environment compared to a single node run. originally detected with the mpich test case (*writeshf90*) Here is a minimal reproducer with MPI_File_read_shared, https://gist.github.com/AboorvaDevarajan/db4797e02ac71f60384a8254ae2862c8 [aboorvad@c712f6n06 io]$ cat hostfile_mpich c712f6n06:2 c712f6n07:2 [aboorvad@c712f6n06 io]$ time mpirun -np 4 --hostfile hostfile_mpich ./read_shared *real 0m14.709s* user 0m27.703s sys 0m1.438s [aboorvad@c712f6n06 io]$ time mpirun -np 4 ./read_shared *real 0m0.247s* user 0m0.153s sys 0m0.616s Is it an expected behaviour? If yes, why? Thanks. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0