November 2015 - discuss - mailman3.cels.anl.gov

Announcing the availability of mpich-3.2
by Kenneth Raffenetti 11 Nov '15

11 Nov '15

The MPICH team is pleased to announce the availability of a new stable release (mpich-3.2). This is the first stable release in the 3.2 series. It contains several new features, enhancements and bug fixes. All production environments are encouraged to upgrade to this release. http://www.mpich.org A list of changes in this release is included at the end of this email. Regards, The MPICH Team =============================================================================== Changes in 3.2 =============================================================================== # Added support for MPI-3.1 features including nonblocking collective I/O, address manipulation routines, thread-safety for MPI initialization, pre-init functionality, and new MPI_T routines to look up variables by name. # Fortran 2008 bindings are enabled by default and fully supported. # Added support for the Mellanox MXM InfiniBand interface. (thanks to Mellanox for the code contribution). # Added support for the Mellanox HCOLL interface for collectives. (thanks to Mellanox for the code contribution). # Significant stability improvements to the MPICH/portals4 implementation. # Completely revamped RMA infrastructure including several scalability improvements, performance improvements, and bug fixes. # Added experimental support for Open Fabrics Interfaces (OFI) version 1.0.0. https://github.com/ofiwg/libfabric (thanks to Intel for code contribution) # The Myrinet MX network module, which had a life cyle from 1.1 till 3.1.2, has now been deleted. # Several other minor bug fixes, memory leak fixes, and code cleanup. A full list of changes is available at the following link: http://git.mpich.org/mpich.git/shortlog/v3.1.3..v3.2 A full list of bugs that have been fixed is available at the following link: https://trac.mpich.org/projects/mpich/query?status=closed&group=resolution&… _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] error spawning processes in mpich-3.2rc1
by Min Si 10 Nov '15

10 Nov '15

Hi Siegmar, We have had a quick fix for the issue in the spawn programs. But the unaligned memory access may exist in every received packet inside MPICH, thus we decided to completely fix it and will add this fix in future release (not mpich-3.2rc2). Attached patch is our quick fix for received eager packets, it solves the issue in your spawn programs. You can patch it to mpich-3.2rc1 for temporary use. cd <mpich-3.2rc1 source dir> gpatch -p1 < unalign_access.patch autoconf -f ./configure <your options> make && make install Please also track the ticket for official fix. Best regards, Min On 11/10/15 3:40 AM, Siegmar Gross wrote: > Hi Min, > > yesterday I installed mpich-3.2rc2 and it still breaks. Are you still > trying to solve the problem? > > > Best regards > > Siegmar > >>>> Min Si<msi(a)il.is.s.u-tokyo.ac.jp> 10/21/15 6:36 PM >>> > Hi Siegmar, > > Thanks for providing us the test machine. We have confirmed this failure > is caused by unaligned memory access inside MPICH. So the failure only > happens on SPARC, which is alignment-sensitive. > > We will fix it. You can track the progress from this ticket. > https://trac.mpich.org/projects/mpich/ticket/2309#ticket > > Because we do not have other SPARC platform, would you mind if we use > your machine for testing during this fix ? > > Best regards, > Min > On 10/12/15 9:24 AM, Siegmar Gross wrote: >> Hi Min, >> >>> It seems you already enabled the most detailed error outputs. We could >>> not think out any clue for now. If you can give us access to your >>> machine, we are glad to help you debug on it. >> Can you send me your email address because I don't want to send >> login data to this list. >> >> >> Kind regards >> >> Siegmar >> >> >>> Min >>> >>> On 10/8/15 12:02 AM, Siegmar Gross wrote: >>>> Hi Min, >>>> >>>> thank you very much for your answer. >>>> >>>>> We cannot reproduce this error on our test machines (Solaris i386, >>>>> Ubuntu x86_64) by using your programs. And unfortunately we do not >>>>> have >>>>> Solaris Sparc machine thus could not verify it. >>>> The programs work fine on my Solaris x86_64 and Linux machines >>>> as well. I only have a problem on Solaris Sparc. >>>> >>>> >>>>> Sometime, it can happen that you need to add "./" in front of the >>>>> program path, could you try it ? >>>>> >>>>> For example, in spawn_master.c MPI: A Message-Passing Interface >>>>> Standard >>>>>> #define SLAVE_PROG "./spawn_slave" >>>> No, it wil not work, because the programs are stored in a >>>> different directory ($HOME/{SunOS, Linux}/{sparc, x86_64}/bin) >>>> which is part of PATH (as well as "."). >>>> >>>> Can I do anything to track the source of the error? >>>> >>>> >>>> Kind regards >>>> >>>> Siegmar >>>> >>>>> Min >>>>> >>>>> On 10/7/15 5:03 AM, Siegmar Gross wrote: >>>>>> Hi, >>>>>> >>>>>> today I've built mpich-3.2rc1 on my machines (Solaris 10 Sparc, >>>>>> Solaris 10 x86_64, and openSUSE Linux 12.1 x86_64) with gcc-5.1.0 >>>>>> and Sun C 5.13. I still get the following errors on my Sparc machine >>>>>> which I'd already reported September 8th. "mpiexec" is aliased to >>>>>> 'mpiexec -genvnone'. It still doesn't matter if I use my cc- or >>>>>> gcc-version of MPICH. >>>>>> >>>>>> >>>>>> tyr spawn 119 mpichversion >>>>>> MPICH Version: 3.2rc1 >>>>>> MPICH Release date: Wed Oct 7 00:00:33 CDT 2015 >>>>>> MPICH Device: ch3:nemesis >>>>>> MPICH configure: --prefix=/usr/local/mpich-3.2_64_cc >>>>>> --libdir=/usr/local/mpich-3.2_64_cc/lib64 >>>>>> --includedir=/usr/local/mpich-3.2_64_cc/include64 CC=cc CXX=CC >>>>>> F77=f77 >>>>>> FC=f95 CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 FCFLAGS=-m64 >>>>>> LDFLAGS=-m64 >>>>>> -L/usr/lib/sparcv9 -R/usr/lib/sparcv9 --enable-fortran=yes >>>>>> --enable-cxx --enable-romio --enable-debuginfo --enable-smpcoll >>>>>> --enable-threads=multiple --with-thread-package=posix --enable-shared >>>>>> MPICH CC: cc -m64 -O2 >>>>>> MPICH CXX: CC -m64 -O2 >>>>>> MPICH F77: f77 -m64 >>>>>> MPICH FC: f95 -m64 -O2 >>>>>> tyr spawn 120 >>>>>> >>>>>> >>>>>> >>>>>> tyr spawn 111 mpiexec -np 1 spawn_master >>>>>> >>>>>> Parent process 0 running on tyr.informatik.hs-fulda.de >>>>>> I create 4 slave processes >>>>>> >>>>>> Fatal error in MPI_Comm_spawn: Unknown error class, error stack: >>>>>> MPI_Comm_spawn(144)...........: MPI_Comm_spawn(cmd="spawn_slave", >>>>>> argv=0, maxprocs=4, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, >>>>>> intercomm=ffffffff7fffde50, errors=0) failed >>>>>> MPIDI_Comm_spawn_multiple(274): >>>>>> MPID_Comm_accept(153).........: >>>>>> MPIDI_Comm_accept(1057).......: >>>>>> MPIR_Bcast_intra(1287)........: >>>>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> tyr spawn 112 mpiexec -np 1 spawn_multiple_master >>>>>> >>>>>> Parent process 0 running on tyr.informatik.hs-fulda.de >>>>>> I create 3 slave processes. >>>>>> >>>>>> Fatal error in MPI_Comm_spawn_multiple: Unknown error class, error >>>>>> stack: >>>>>> MPI_Comm_spawn_multiple(162)..: MPI_Comm_spawn_multiple(count=2, >>>>>> cmds=ffffffff7fffde08, argvs=ffffffff7fffddf8, >>>>>> maxprocs=ffffffff7fffddf0, infos=ffffffff7fffdde8, root=0, >>>>>> MPI_COMM_WORLD, intercomm=ffffffff7fffdde4, errors=0) failed >>>>>> MPIDI_Comm_spawn_multiple(274): >>>>>> MPID_Comm_accept(153).........: >>>>>> MPIDI_Comm_accept(1057).......: >>>>>> MPIR_Bcast_intra(1287)........: >>>>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> tyr spawn 113 mpiexec -np 1 spawn_intra_comm >>>>>> Parent process 0: I create 2 slave processes >>>>>> Fatal error in MPI_Comm_spawn: Unknown error class, error stack: >>>>>> MPI_Comm_spawn(144)...........: >>>>>> MPI_Comm_spawn(cmd="spawn_intra_comm", >>>>>> argv=0, maxprocs=2, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, >>>>>> intercomm=ffffffff7fffded4, errors=0) failed >>>>>> MPIDI_Comm_spawn_multiple(274): >>>>>> MPID_Comm_accept(153).........: >>>>>> MPIDI_Comm_accept(1057).......: >>>>>> MPIR_Bcast_intra(1287)........: >>>>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>>>> tyr spawn 114 >>>>>> >>>>>> >>>>>> I would be grateful if somebody can fix the problem. Thank you very >>>>>> much for any help in advance. I've attached my programs. Please let >>>>>> me know if you need anything else. >>>>>> >>>>>> >>>>>> Kind regards >>>>>> >>>>>> Siegmar >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailinglistdiscuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>>> _______________________________________________ >>>>> discuss mailing listdiscuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> >>>> _______________________________________________ >>>> discuss mailinglistdiscuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing listdiscuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> _______________________________________________ >> discuss mailing listdiscuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Bug (?) report: potential division by zero in ADIOI_LUSTRE_Docollect
by Kokron, Daniel S. (ARC-606.2)[Computer Sciences Corporation] 10 Nov '15

10 Nov '15

Constantine, Let me know if you want help extracting a reproducer. Daniel Kokron NASA Ames (ARC-TN) SciCon group ________________________________________ From: Michael Raymond [mraymond(a)sgi.com] Sent: Tuesday, November 10, 2015 07:33 To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Bug (?) report: potential division by zero in ADIOI_LUSTRE_Docollect Constantine, as soon as you can reproduce this, please have NASA get in touch with SGI Customer Service and we'll jump right on it. On 11/09/2015 05:14 PM, Rob Latham wrote: >> >> Some context: I am debugging a failure of PISM [1] on NASA >> Pleiades [2], which uses the SGI MPI implementation, which also uses >> ROMIO. PISM crashed with SIGFPE in ADIOI_LUSTRE_Docollect deep inside >> HDF5 and NetCDF. I am not asking for help with this [3] here; I will >> contact NASA's support as soon as I have a way of reproducing the >> issue outside of PISM. > > I wonder if Michael Raymond is on this list? > > ==rob > -- Michael A. Raymond SGI MPT Team Leader 1 (651) 683-7523 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Bug (?) report: potential division by zero in ADIOI_LUSTRE_Docollect
by Michael Raymond 10 Nov '15

10 Nov '15

Constantine, as soon as you can reproduce this, please have NASA get in touch with SGI Customer Service and we'll jump right on it. On 11/09/2015 05:14 PM, Rob Latham wrote: >> >> Some context: I am debugging a failure of PISM [1] on NASA >> Pleiades [2], which uses the SGI MPI implementation, which also uses >> ROMIO. PISM crashed with SIGFPE in ADIOI_LUSTRE_Docollect deep inside >> HDF5 and NetCDF. I am not asking for help with this [3] here; I will >> contact NASA's support as soon as I have a way of reproducing the >> issue outside of PISM. > > I wonder if Michael Raymond is on this list? > > ==rob > -- Michael A. Raymond SGI MPT Team Leader 1 (651) 683-7523 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.1.4 Behavior
by Daniel Buchwieser da Silva 10 Nov '15

10 Nov '15

Thanks a lot. I changed my /etc/hosts config and its work fine now. I also removed the confg files from the old version of mpich (/usr/local/etc) too. The hosts files was configured with the host name for loopback interface, I commented the entry and ran rthe test again. Regards for all Daniel Buchwieser da Silva Analista de Informática U.O. Sistemas de Informação e Telecomunicação-DSI Fone: +55 +55 (48) 3221-7311 danielbs(a)tractebelenergia.com.br http://www.tractebelenergia.com.br Antes de imprimir, pense em sua responsabilidade e compromisso com o MEIO AMBIENTE. Before printing, think about your ENVIRONMENTAL responsibility. De: "Balaji, Pavan" <balaji(a)anl.gov> Para: "discuss(a)mpich.org" <discuss(a)mpich.org> Data: 09/11/2015 20:29 Assunto: Re: [mpich-discuss] MPICH 3.1.4 Behavior This looks like either a firewall issue or an /etc/hosts setup issue. Did you look at our FAQ page? https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions -- Pavan > On Nov 9, 2015, at 4:00 AM, Daniel Buchwieser da Silva <danielbs(a)tractebelenergia.com.br> wrote: > > Good morning > > I have a problem with the version 3.1.4 MPICH when I try to execute a program across different machines with multiples processors. The error is showed below someone have ever seen this error or resolved then? > > Fatal error in MPI_Bcast: Other MPI error, error stack: > MPI_Bcast(786)..............................: MPI_Bcast(buf=0x1d2af88, count=1, MPI_INTEGER, root=0, MPI_COMM_WORLD) failed > MPIR_Bcast(230).............................: > MPIC_Send(39)...............................: > MPIC_Wait(270)..............................: > MPIDI_CH3i_Progress_wait(215)...............: an error occurred while handling an event returned by MPIDU_Sock_Wait() > MPIDI_CH3I_Progress_handle_sock_event(779)..: > MPIDI_CH3_Sockconn_handle_connect_event(608): [ch3:sock] failed to connnect to remote process > MPIDU_Socki_handle_connect(795).............: connection failure (set=0,sock=1,errno=111:Connection refused)[cli_0]: aborting job: > Fatal error in MPI_Bcast: Other MPI error, error stack: > MPI_Bcast(786)..............................: MPI_Bcast(buf=0x1d2af88, count=1, MPI_INTEGER, root=0, MPI_COMM_WORLD) failed > MPIR_Bcast(230).............................: > MPIC_Send(39)...............................: > MPIC_Wait(270)..............................: > MPIDI_CH3i_Progress_wait(215)...............: an error occurred while handling an event returned by MPIDU_Sock_Wait() > MPIDI_CH3I_Progress_handle_sock_event(779)..: > MPIDI_CH3_Sockconn_handle_connect_event(608): [ch3:sock] failed to connnect to remote process > MPIDU_Socki_handle_connect(795).............: connection failure (set=0,sock=1,errno=111:Connection refused) > > Daniel Buchwieser da Silva > Analista de Informática > U.O. Sistemas de Informação e Telecomunicação-DSI > Fone: +55 +55 (48) 3221-7311 > danielbs(a)tractebelenergia.com.br > http://www.tractebelenergia.com.br > Antes de imprimir, pense em sua responsabilidade e compromisso com o MEIO AMBIENTE. > Before printing, think about your ENVIRONMENTAL responsibility. > > > > Aviso Legal: > Esta mensagem é para uso exclusivo do(s) destinatário(s) indicado(s) e pode conter informações protegidas por segredo profissional, por direitos do autor ou por outras normas legais. Se você a recebeu por engano, por favor, comunique-nos mediante resposta e a elimine de seus sistemas. É proibido copiar ou revelar o seu conteúdo a qualquer outra pessoa. A confidencialidade e o segredo profissional legal não são diminuídos ou invalidados por qualquer envio equivocado desta mensagem. Todas as mensagens enviadas a/por esta Companhia podem ser controladas e monitoradas para garantir o cumprimento das políticas internas, para proteger os interesses da Companhia e para eliminar softwares mal intencionados (malware). No entanto, as mensagens eletrônicas não são seguras e podem conter erros, ser interceptadas, modificadas, perdidas, eliminadas ou conter algum tipo de vírus. Qualquer pessoa que se comunica com nossa Companhia por correio eletrônico aceita esses riscos. As delegações de poderes e de autoridade devem ser verificadas e estão disponíveis mediante solicitação. > > > Disclaimer: > This message is for the sole use of the intended recipient(s) and may have privileged information or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply and delete it from your systems. It is forbidden to copy it or disclose its content to anyone. Any confidentiality or legal professional privilege is not waived or lost by any mistaken delivery of the message. All messages sent to and from this company may be monitored to ensure compliance with internal policies, to protect the company's interests and to remove potential malware. However, electronic messages are not secure nor error free as they can be intercepted, amended, lost or deleted, or contain viruses. Anyone who communicates with our company by electronic messaging accepts these risks. Delegations of power and authority have to be verified and are available upon request. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss Aviso Legal: Esta mensagem é para uso exclusivo do(s) destinatário(s) indicado(s) e pode conter informações protegidas por segredo profissional, por direitos do autor ou por outras normas legais. Se você a recebeu por engano, por favor, comunique-nos mediante resposta e a elimine de seus sistemas. É proibido copiar ou revelar o seu conteúdo a qualquer outra pessoa. A confidencialidade e o segredo profissional legal não são diminuídos ou invalidados por qualquer envio equivocado desta mensagem. Todas as mensagens enviadas a/por esta Companhia podem ser controladas e monitoradas para garantir o cumprimento das políticas internas, para proteger os interesses da Companhia e para eliminar softwares mal intencionados (malware). No entanto, as mensagens eletrônicas não são seguras e podem conter erros, ser interceptadas, modificadas, perdidas, eliminadas ou conter algum tipo de vírus. Qualquer pessoa que se comunica com nossa Companhia por correio eletrônico aceita esses riscos. As delegações de poderes e de autoridade devem ser verificadas e estão disponíveis mediante solicitação. Disclaimer: This message is for the sole use of the intended recipient(s) and may have privileged information or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply and delete it from your systems. It is forbidden to copy it or disclose its content to anyone. Any confidentiality or legal professional privilege is not waived or lost by any mistaken delivery of the message. All messages sent to and from this company may be monitored to ensure compliance with internal policies, to protect the company's interests and to remove potential malware. However, electronic messages are not secure nor error free as they can be intercepted, amended, lost or deleted, or contain viruses. Anyone who communicates with our company by electronic messaging accepts these risks. Delegations of power and authority have to be verified and are available upon request. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Bug (?) report: potential division by zero in ADIOI_LUSTRE_Docollect
by Rob Latham 09 Nov '15

09 Nov '15

On 10/30/2015 03:36 PM, Constantine Khroulev wrote: > Dear MPICH developers, > > I am writing to you to report what I think is a bug in ADIO (which, if > I understand it correctly, is a part of ROMIO, which is a part of > MPICH). > > The function int ADIOI_LUSTRE_Docollect(ADIO_File, int, ADIO_Offset *, > int) defined in src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c > (MPICH version 3.1.4 and several earlier versions) contains an > unprotected division: > > /* estimate average req_size */ > avg_req_size = (int)(total_req_size / total_access_count); > > I suggest adding an if statement protecting from division by zero and > stopping (if appropriate). thanks for taking a closer look at the lustre code. Halim and I chatted a bit about this. We are frozen for the upcoming release, but once we re-open I'll apply the patch below: % git diff src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c diff --git a/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c b/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c index cd552829..e7901212 100644 --- a/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c +++ b/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c @@ -309,9 +309,14 @@ int ADIOI_LUSTRE_Docollect(ADIO_File fd, int contig_access_count, MPI_Allreduce(&req_size, &total_req_size, 1, MPI_LONG_LONG_INT, MPI_SUM, fd->comm); MPI_Allreduce(&contig_access_count, &total_access_count, 1, MPI_INT, MPI_SUM, - fd->comm); - /* estimate average req_size */ - avg_req_size = (int)(total_req_size / total_access_count); + fd->comm); + /* avoid possible divide-by-zero) */ + if (!total_access_count) { + /* estimate average req_size */ + avg_req_size = (int)(total_req_size / total_access_count); + } else { + avg_req_size = 0; + } /* get hint of big_req_size */ big_req_size = fd->hints->fs_hints.lustre.coll_threshold; /* Don't perform collective I/O if there are big requests */ > > Some context: I am debugging a failure of PISM [1] on NASA > Pleiades [2], which uses the SGI MPI implementation, which also uses > ROMIO. PISM crashed with SIGFPE in ADIOI_LUSTRE_Docollect deep inside > HDF5 and NetCDF. I am not asking for help with this [3] here; I will > contact NASA's support as soon as I have a way of reproducing the > issue outside of PISM. I wonder if Michael Raymond is on this list? ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.1.4 Behavior
by Balaji, Pavan 09 Nov '15

09 Nov '15

This looks like either a firewall issue or an /etc/hosts setup issue. Did you look at our FAQ page? https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions -- Pavan > On Nov 9, 2015, at 4:00 AM, Daniel Buchwieser da Silva <danielbs(a)tractebelenergia.com.br> wrote: > > Good morning > > I have a problem with the version 3.1.4 MPICH when I try to execute a program across different machines with multiples processors. The error is showed below someone have ever seen this error or resolved then? > > Fatal error in MPI_Bcast: Other MPI error, error stack: > MPI_Bcast(786)..............................: MPI_Bcast(buf=0x1d2af88, count=1, MPI_INTEGER, root=0, MPI_COMM_WORLD) failed > MPIR_Bcast(230).............................: > MPIC_Send(39)...............................: > MPIC_Wait(270)..............................: > MPIDI_CH3i_Progress_wait(215)...............: an error occurred while handling an event returned by MPIDU_Sock_Wait() > MPIDI_CH3I_Progress_handle_sock_event(779)..: > MPIDI_CH3_Sockconn_handle_connect_event(608): [ch3:sock] failed to connnect to remote process > MPIDU_Socki_handle_connect(795).............: connection failure (set=0,sock=1,errno=111:Connection refused)[cli_0]: aborting job: > Fatal error in MPI_Bcast: Other MPI error, error stack: > MPI_Bcast(786)..............................: MPI_Bcast(buf=0x1d2af88, count=1, MPI_INTEGER, root=0, MPI_COMM_WORLD) failed > MPIR_Bcast(230).............................: > MPIC_Send(39)...............................: > MPIC_Wait(270)..............................: > MPIDI_CH3i_Progress_wait(215)...............: an error occurred while handling an event returned by MPIDU_Sock_Wait() > MPIDI_CH3I_Progress_handle_sock_event(779)..: > MPIDI_CH3_Sockconn_handle_connect_event(608): [ch3:sock] failed to connnect to remote process > MPIDU_Socki_handle_connect(795).............: connection failure (set=0,sock=1,errno=111:Connection refused) > > Daniel Buchwieser da Silva > Analista de Informática > U.O. Sistemas de Informação e Telecomunicação-DSI > Fone: +55 +55 (48) 3221-7311 > danielbs(a)tractebelenergia.com.br > http://www.tractebelenergia.com.br > Antes de imprimir, pense em sua responsabilidade e compromisso com o MEIO AMBIENTE. > Before printing, think about your ENVIRONMENTAL responsibility. > > > > Aviso Legal: > Esta mensagem é para uso exclusivo do(s) destinatário(s) indicado(s) e pode conter informações protegidas por segredo profissional, por direitos do autor ou por outras normas legais. Se você a recebeu por engano, por favor, comunique-nos mediante resposta e a elimine de seus sistemas. É proibido copiar ou revelar o seu conteúdo a qualquer outra pessoa. A confidencialidade e o segredo profissional legal não são diminuídos ou invalidados por qualquer envio equivocado desta mensagem. Todas as mensagens enviadas a/por esta Companhia podem ser controladas e monitoradas para garantir o cumprimento das políticas internas, para proteger os interesses da Companhia e para eliminar softwares mal intencionados (malware). No entanto, as mensagens eletrônicas não são seguras e podem conter erros, ser interceptadas, modificadas, perdidas, eliminadas ou conter algum tipo de vírus. Qualquer pessoa que se comunica com nossa Companhia por correio eletrônico aceita esses riscos. As delegações de poderes e de autoridade devem ser verificadas e estão disponíveis mediante solicitação. > > > Disclaimer: > This message is for the sole use of the intended recipient(s) and may have privileged information or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply and delete it from your systems. It is forbidden to copy it or disclose its content to anyone. Any confidentiality or legal professional privilege is not waived or lost by any mistaken delivery of the message. All messages sent to and from this company may be monitored to ensure compliance with internal policies, to protect the company's interests and to remove potential malware. However, electronic messages are not secure nor error free as they can be intercepted, amended, lost or deleted, or contain viruses. Anyone who communicates with our company by electronic messaging accepts these risks. Delegations of power and authority have to be verified and are available upon request. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [Fwd: MPI_File_read_at_all external32 bug in MV2-2.1]
by Rob Latham 09 Nov '15

09 Nov '15

On 11/09/2015 03:22 PM, Adam T. Moody wrote: > Hi Rob, > Ok, thanks for the background. I'll put together a simple reproducer > for the particular case that I've got. I'll send that a little later. > > The bug that OpenMPI found is the same kind of bug but in a different > routine (read instead of read_at_all). My guess is that this is likely > a cut-and-paste bug, so it'd be a good idea to check the other MPI IO > routines for this problem. I took a crack at reducing the cut and paste when I first merged the patch but clearly more should be done... and now that I know there's a user out there, I'll clean it up. My usual advice is to use something like Parallel-Netcdf or HDF5, which have portability like external32 but also have a self-describing file format. A library option is not an answer for you? ==rob > -Adam > > > Rob Latham wrote: > >> >> >> On 11/06/2015 07:13 PM, Balaji, Pavan wrote: >> >>> Adam, >>> >>> Not yet. We were working heads-down on the mpich-3.2 release. >>> >>> I'm adding the mpich discuss list to cc, and poking @robl as well. Do >>> you have a test program that reproduces the error? (sorry if you >>> already sent it, I'm still catching up on the pending issues). >>> >> >> adam, thanks for reporting. Since you're the first person I've ever >> met who cared about external32 support, let me give you a bit of >> background. >> >> Our External32 support came from Intel, and we know about one bug : if >> the externa32 representation differs from the memory size (MPI_LONG >> and MPI_AINT are perhaps the two most prominent examples) >> >> - http://trac.mpich.org/projects/mpich/ticket/1754 (but look for >> 'goodell' and 'robl'. 'jhammond' comments are less relevant for this >> discussion) >> >> - http://trac.mpich.org/projects/mpich/ticket/1755 >> >> We do have an external32 test case, but it's pretty simple, and >> independent i/o only. >> >> OpenMPI found a bug in our external32 code, which sounds related to >> your problem: >> >> http://git.mpich.org/mpich.git/commitdiff/53f11fd73b2f9aa7b994e78ed05e9ae74… >> >> >> So you can see why Pavan was hoping you had a test case: it's taken a >> few tries to get it right. >> >> ==rob >> >>> -- Pavan >>> >>>> On Nov 6, 2015, at 2:10 PM, Adam T. Moody <moody20(a)llnl.gov> wrote: >>>> >>>> Hi Pavan, >>>> Did you have a chance to look into this in MPICH? >>>> >>>> I'm pretty certain there is a lurking ROMIO bug here. >>>> -Adam >>>> >>>> >>>> Adam T. Moody wrote: >>>> >>>>> Hi Pavan and Howard, >>>>> FYI, it looks this same bug is in MPICH-3.2rc1 and Open MPI-1.10.0. >>>>> >>>>> I guess no one uses collective I/O with external32 :-) >>>>> -Adam >>>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> >>>>> Subject: >>>>> MPI_File_read_at_all external32 bug in MV2-2.1 >>>>> From: >>>>> "Adam T. Moody" <moody20(a)llnl.gov> >>>>> Date: >>>>> Fri, 30 Oct 2015 11:48:35 -0700 >>>>> To: >>>>> "mvapich-discuss(a)cse.ohio-state.edu" >>>>> <mvapich-discuss(a)cse.ohio-state.edu> >>>>> >>>>> To: >>>>> "mvapich-discuss(a)cse.ohio-state.edu" >>>>> <mvapich-discuss(a)cse.ohio-state.edu> >>>>> >>>>> >>>>> Hello MVAPICH team, >>>>> I've hit a bug in MPI_File_read_at_all in MVAPICH2-2.1. I have an >>>>> application that reads and writes files in external32 format. It >>>>> writes the file just fine, but it throws the following error when >>>>> reading the file back: >>>>> >>>>> internal ABORT - process 0 >>>>> srun: error: rzmerl2: task 0: Exited with exit code 1 >>>>> Assertion failed in file >>>>> src/mpid/common/datatype/mpid_ext32_segment.c at line 277: FALSE >>>>> memcpy argument memory ranges overlap, dst_=0x2aaab5c160a8 >>>>> src_=0x2aaab5c160a8 len_=176 >>>>> >>>>> Above, MPI is trying to do a memcpy where the source and >>>>> destination buffer are the same address. Looking through the code >>>>> for MVAPICH2-2.1, the problem seems to be at line 132 in >>>>> src/mpi/romio/mpi-io/read_all.c: >>>>> >>>>> if (e32_buf != NULL) { >>>>> error_code = MPIU_read_external32_conversion_fn(xbuf, datatype, >>>>> count, e32_buf); >>>>> ADIOI_Free(e32_buf); >>>>> } >>>>> >>>>> I think the fix is to change "xbuf" above to "buf" as it is in >>>>> read.c below: >>>>> >>>>> if (e32_buf != NULL) { >>>>> error_code = MPIU_read_external32_conversion_fn(buf, datatype, >>>>> count, e32_buf); >>>>> ADIOI_Free(e32_buf); >>>>> } >>>>> >>>>> When in external32 mode, xbuf == e32_buf, which acts as a temporary >>>>> buffer in which to read the data. The code is then meant to unpack >>>>> and convert the data from the temporary buffer into the user buffer >>>>> at buf. >>>>> >>>>> It's probably worth checking the other external32 code paths to >>>>> look for similar bugs. >>>>> Thanks, >>>>> -Adam >>>>> >>>> >>> >> > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] [Fwd: MPI_File_read_at_all external32 bug in MV2-2.1]
by Rob Latham 09 Nov '15

09 Nov '15

On 11/06/2015 07:13 PM, Balaji, Pavan wrote: > Adam, > > Not yet. We were working heads-down on the mpich-3.2 release. > > I'm adding the mpich discuss list to cc, and poking @robl as well. Do you have a test program that reproduces the error? (sorry if you already sent it, I'm still catching up on the pending issues). > adam, thanks for reporting. Since you're the first person I've ever met who cared about external32 support, let me give you a bit of background. Our External32 support came from Intel, and we know about one bug : if the externa32 representation differs from the memory size (MPI_LONG and MPI_AINT are perhaps the two most prominent examples) - http://trac.mpich.org/projects/mpich/ticket/1754 (but look for 'goodell' and 'robl'. 'jhammond' comments are less relevant for this discussion) - http://trac.mpich.org/projects/mpich/ticket/1755 We do have an external32 test case, but it's pretty simple, and independent i/o only. OpenMPI found a bug in our external32 code, which sounds related to your problem: http://git.mpich.org/mpich.git/commitdiff/53f11fd73b2f9aa7b994e78ed05e9ae74… So you can see why Pavan was hoping you had a test case: it's taken a few tries to get it right. ==rob > -- Pavan > >> On Nov 6, 2015, at 2:10 PM, Adam T. Moody <moody20(a)llnl.gov> wrote: >> >> Hi Pavan, >> Did you have a chance to look into this in MPICH? >> >> I'm pretty certain there is a lurking ROMIO bug here. >> -Adam >> >> >> Adam T. Moody wrote: >> >>> Hi Pavan and Howard, >>> FYI, it looks this same bug is in MPICH-3.2rc1 and Open MPI-1.10.0. >>> >>> I guess no one uses collective I/O with external32 :-) >>> -Adam >>> >>> ------------------------------------------------------------------------ >>> >>> Subject: >>> MPI_File_read_at_all external32 bug in MV2-2.1 >>> From: >>> "Adam T. Moody" <moody20(a)llnl.gov> >>> Date: >>> Fri, 30 Oct 2015 11:48:35 -0700 >>> To: >>> "mvapich-discuss(a)cse.ohio-state.edu" <mvapich-discuss(a)cse.ohio-state.edu> >>> >>> To: >>> "mvapich-discuss(a)cse.ohio-state.edu" <mvapich-discuss(a)cse.ohio-state.edu> >>> >>> >>> Hello MVAPICH team, >>> I've hit a bug in MPI_File_read_at_all in MVAPICH2-2.1. I have an application that reads and writes files in external32 format. It writes the file just fine, but it throws the following error when reading the file back: >>> >>> internal ABORT - process 0 >>> srun: error: rzmerl2: task 0: Exited with exit code 1 >>> Assertion failed in file src/mpid/common/datatype/mpid_ext32_segment.c at line 277: FALSE >>> memcpy argument memory ranges overlap, dst_=0x2aaab5c160a8 src_=0x2aaab5c160a8 len_=176 >>> >>> Above, MPI is trying to do a memcpy where the source and destination buffer are the same address. Looking through the code for MVAPICH2-2.1, the problem seems to be at line 132 in src/mpi/romio/mpi-io/read_all.c: >>> >>> if (e32_buf != NULL) { >>> error_code = MPIU_read_external32_conversion_fn(xbuf, datatype, >>> count, e32_buf); >>> ADIOI_Free(e32_buf); >>> } >>> >>> I think the fix is to change "xbuf" above to "buf" as it is in read.c below: >>> >>> if (e32_buf != NULL) { >>> error_code = MPIU_read_external32_conversion_fn(buf, datatype, >>> count, e32_buf); >>> ADIOI_Free(e32_buf); >>> } >>> >>> When in external32 mode, xbuf == e32_buf, which acts as a temporary buffer in which to read the data. The code is then meant to unpack and convert the data from the temporary buffer into the user buffer at buf. >>> >>> It's probably worth checking the other external32 code paths to look for similar bugs. >>> Thanks, >>> -Adam >>> >> > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] MPICH 3.1.4 Behavior
by Oden, Lena 09 Nov '15

09 Nov '15

Daniel, can you please send me a short Demo-program to reproduce the error? For this, I also need your mpich configuration. How many processes and nodes are you using? Lena On 09 Nov 2015, at 04:00, Daniel Buchwieser da Silva <danielbs(a)tractebelenergia.com.br<mailto:[email protected]>> wrote: Good morning I have a problem with the version 3.1.4 MPICH when I try to execute a program across different machines with multiples processors. The error is showed below someone have ever seen this error or resolved then? Fatal error in MPI_Bcast: Other MPI error, error stack: MPI_Bcast(786)..............................: MPI_Bcast(buf=0x1d2af88, count=1, MPI_INTEGER, root=0, MPI_COMM_WORLD) failed MPIR_Bcast(230).............................: MPIC_Send(39)...............................: MPIC_Wait(270)..............................: MPIDI_CH3i_Progress_wait(215)...............: an error occurred while handling an event returned by MPIDU_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(779)..: MPIDI_CH3_Sockconn_handle_connect_event(608): [ch3:sock] failed to connnect to remote process MPIDU_Socki_handle_connect(795).............: connection failure (set=0,sock=1,errno=111:Connection refused)[cli_0]: aborting job: Fatal error in MPI_Bcast: Other MPI error, error stack: MPI_Bcast(786)..............................: MPI_Bcast(buf=0x1d2af88, count=1, MPI_INTEGER, root=0, MPI_COMM_WORLD) failed MPIR_Bcast(230).............................: MPIC_Send(39)...............................: MPIC_Wait(270)..............................: MPIDI_CH3i_Progress_wait(215)...............: an error occurred while handling an event returned by MPIDU_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(779)..: MPIDI_CH3_Sockconn_handle_connect_event(608): [ch3:sock] failed to connnect to remote process MPIDU_Socki_handle_connect(795).............: connection failure (set=0,sock=1,errno=111:Connection refused) Daniel Buchwieser da Silva Analista de Informática U.O. Sistemas de Informação e Telecomunicação-DSI Fone: +55 +55 (48) 3221-7311 danielbs(a)tractebelenergia.com.br<mailto:[email protected]> http://www.tractebelenergia.com.br<http://www.tractebelenergia.com.br/> Antes de imprimir, pense em sua responsabilidade e compromisso com o MEIO AMBIENTE. Before printing, think about your ENVIRONMENTAL responsibility. Aviso Legal: Esta mensagem é para uso exclusivo do(s) destinatário(s) indicado(s) e pode conter informações protegidas por segredo profissional, por direitos do autor ou por outras normas legais. Se você a recebeu por engano, por favor, comunique-nos mediante resposta e a elimine de seus sistemas. É proibido copiar ou revelar o seu conteúdo a qualquer outra pessoa. A confidencialidade e o segredo profissional legal não são diminuídos ou invalidados por qualquer envio equivocado desta mensagem. Todas as mensagens enviadas a/por esta Companhia podem ser controladas e monitoradas para garantir o cumprimento das políticas internas, para proteger os interesses da Companhia e para eliminar softwares mal intencionados (malware). No entanto, as mensagens eletrônicas não são seguras e podem conter erros, ser interceptadas, modificadas, perdidas, eliminadas ou conter algum tipo de vírus. Qualquer pessoa que se comunica com nossa Companhia por correio eletrônico aceita esses riscos. As delegações de poderes e de autoridade devem ser verificadas e estão disponíveis mediante solicitação. Disclaimer: This message is for the sole use of the intended recipient(s) and may have privileged information or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply and delete it from your systems. It is forbidden to copy it or disclose its content to anyone. Any confidentiality or legal professional privilege is not waived or lost by any mistaken delivery of the message. All messages sent to and from this company may be monitored to ensure compliance with internal policies, to protect the company's interests and to remove potential malware. However, electronic messages are not secure nor error free as they can be intercepted, amended, lost or deleted, or contain viruses. Anyone who communicates with our company by electronic messaging accepts these risks. Delegations of power and authority have to be verified and are available upon request. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0