- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] possible thread bug with MPI_Reduce/MPI_Allreduce
by Zhou, Hui 14 Jul '23

14 Jul '23

Glad I could help. Hui ________________________________ From: Burlen Loring <burlen.loring(a)gmail.com> Sent: Friday, July 14, 2023 12:31 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] possible thread bug with MPI_Reduce/MPI_Allreduce Hi Hui, The wording in the documentation of MPI_Init_thread, and specifically the part that says with MPI_THREAD_MULTIPLE that "multiple threads may call MPI at once with no restrictions", had me thinking along the wrong lines. Following your suggestions, I rewrote my code to give each thread exclusive access to one communicator from a small pool of communicators. This is working well and out performed Iallreduce/Ireduce approach. Thanks so much for the help Burlen On 7/12/23 11:55, Zhou, Hui wrote: Hi Burlen, You are not allowed to issue collectives concurrently on the same communicator. You can use non-blocking collectives, i.e. MPI_Ireduce and MPI_Iallreduce to achieve the overlapping. You still need serialize the issue of MPI_Ireduce/MPI_Iallreduce, but you don't need mutex to wait for the request to complete in concurrent threads. -- Hui ________________________________ From: Burlen Loring via discuss <discuss(a)mpich.org><mailto:[email protected]> Sent: Wednesday, July 12, 2023 1:29 PM To: <discuss(a)mpich.org><mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: Burlen Loring <burlen.loring(a)gmail.com><mailto:[email protected]> Subject: [mpich-discuss] possible thread bug with MPI_Reduce/MPI_Allreduce Hi All, I'm using MPICH 4.0.2 on Fedora 37 from the package manager for development. From an MPI parallel simulation I'm spawning a thread that does a number of reductions (MPI_Allreduce and MPI_Reduce). MPI_IN_PLACE option is used. The results are written with POSIX I/O from rank 0. The simulation continues, and can launch the next set of reductions before the previous ones completed. I have called MPI_Init_thread and requested and received MPI_THREAD_MULTIPLE support. However, when multiple threads overlap (in test runs 3-4 threads running concurrently) both MPI_Allreduce and MPI_Reduce calls can produce incorrect results. If instead I serialize the threads, by waiting on them before returning to the simulation, the results are correct. Also, if I use a mutex around my MPI_Allreduce/Reduce sections, the results are correct. I think that MPI_Reduce/Allreduce is not thread safe. I was wondering if this is a known issue? Could it be a mpich build/configure setting not set correctly by the Fedora package maintainer ? Thanks Burlen

1 0

Re: [mpich-discuss] possible thread bug with MPI_Reduce/MPI_Allreduce
by Zhou, Hui 12 Jul '23

12 Jul '23

Hi Burlen, You are not allowed to issue collectives concurrently on the same communicator. You can use non-blocking collectives, i.e. MPI_Ireduce and MPI_Iallreduce to achieve the overlapping. You still need serialize the issue of MPI_Ireduce/MPI_Iallreduce, but you don't need mutex to wait for the request to complete in concurrent threads. -- Hui ________________________________ From: Burlen Loring via discuss <discuss(a)mpich.org> Sent: Wednesday, July 12, 2023 1:29 PM To: <discuss(a)mpich.org> <discuss(a)mpich.org> Cc: Burlen Loring <burlen.loring(a)gmail.com> Subject: [mpich-discuss] possible thread bug with MPI_Reduce/MPI_Allreduce Hi All, I'm using MPICH 4.0.2 on Fedora 37 from the package manager for development. From an MPI parallel simulation I'm spawning a thread that does a number of reductions (MPI_Allreduce and MPI_Reduce). MPI_IN_PLACE option is used. The results are written with POSIX I/O from rank 0. The simulation continues, and can launch the next set of reductions before the previous ones completed. I have called MPI_Init_thread and requested and received MPI_THREAD_MULTIPLE support. However, when multiple threads overlap (in test runs 3-4 threads running concurrently) both MPI_Allreduce and MPI_Reduce calls can produce incorrect results. If instead I serialize the threads, by waiting on them before returning to the simulation, the results are correct. Also, if I use a mutex around my MPI_Allreduce/Reduce sections, the results are correct. I think that MPI_Reduce/Allreduce is not thread safe. I was wondering if this is a known issue? Could it be a mpich build/configure setting not set correctly by the Fedora package maintainer ? Thanks Burlen

1 0

possible thread bug with MPI_Reduce/MPI_Allreduce
by Burlen Loring 12 Jul '23

12 Jul '23

Hi All, I'm using MPICH 4.0.2 on Fedora 37 from the package manager for development. From an MPI parallel simulation I'm spawning a thread that does a number of reductions (MPI_Allreduce and MPI_Reduce). MPI_IN_PLACE option is used. The results are written with POSIX I/O from rank 0. The simulation continues, and can launch the next set of reductions before the previous ones completed. I have called MPI_Init_thread and requested and received MPI_THREAD_MULTIPLE support. However, when multiple threads overlap (in test runs 3-4 threads running concurrently) both MPI_Allreduce and MPI_Reduce calls can produce incorrect results. If instead I serialize the threads, by waiting on them before returning to the simulation, the results are correct. Also, if I use a mutex around my MPI_Allreduce/Reduce sections, the results are correct. I think that MPI_Reduce/Allreduce is not thread safe. I was wondering if this is a known issue? Could it be a mpich build/configure setting not set correctly by the Fedora package maintainer ? Thanks Burlen

1 0

Re: [mpich-discuss] MPI_Init fail
by Raffenetti, Ken 26 Jun '23

26 Jun '23

The looks to be an error from a fairly old version of MPICH. Have you tried installing the latest version (4.1.2) available on our website or Github? Ken From: Nazmul Haque via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Sunday, June 25, 2023 at 11:38 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Nazmul Haque <shadhin873(a)outlook.com> Subject: [mpich-discuss] MPI_Init fail Hello, I have installed mpich2 and tried to run an executable with 'mpiexec -n 8 lmp -in .....' but it shows error, replying that mpi_init is unable to connect to the local host. How can I solve this? I have also attached the image of the error. Thank you for your time. Regards, Nazmul

1 0

MPI_Init fail
by Nazmul Haque 25 Jun '23

25 Jun '23

Hello, I have installed mpich2 and tried to run an executable with 'mpiexec -n 8 lmp -in .....' but it shows error, replying that mpi_init is unable to connect to the local host. How can I solve this? I have also attached the image of the error. Thank you for your time. Regards, Nazmul

1 0

Re: [mpich-discuss] Issue in MPICH while submitting jobs through slurm in NONMEM application
by Zhou, Hui 15 Jun '23

15 Jun '23

Hi Thomas, The assertion error means packet corruption. It's not clear what could be the cause unless you can provide a reproducer. Anyway, mpich-3.2.1 is quite old. My first suggestion would be try a newer mpich see if the error persists. -- Hui Zhou ________________________________ From: Thomas Jayaseelan-External via discuss <discuss(a)mpich.org> Sent: Thursday, June 15, 2023 9:58 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thomas Jayaseelan-External <thomas.jayaseelan(a)regeneron.com>; Sundaresh Krishnasamy-External <sundaresh.krishnasam(a)regeneron.com>; Hariram Jayaram-External <hariram.jayaram(a)regeneron.com> Subject: Re: [mpich-discuss] Issue in MPICH while submitting jobs through slurm in NONMEM application Hi All, It would helpful if you could help me on the below issue that we face in our application using MPI. Best Regards, Thomas From: Thomas Jayaseelan-External Sent: Thursday, June 15, 2023 10:58 AM To: discuss(a)mpich.org Cc: Sundaresh Krishnasamy-External <sundaresh.krishnasam(a)regeneron.com>; Hariram Jayaram-External <hariram.jayaram(a)regeneron.com> Subject: Issue in MPICH while submitting jobs through slurm in NONMEM application Hi Team, This is Thomas, I am part of HPCOPs team in Regeneron Pharmaceuticals company. We build and support the HPC cluster infrastructure for the business as per their requirements. I have reached out to you to get help on an issue that we are currently facing with MPI. It would be great if you could help us in getting a solution to it. Nonmem is the application in which users submit the jobs through CLI, it is a CLI based application. When user tries to run job with more no. of cores the job runs for 10 to 15 hours and then stops intermittently. Please find the below error message that we get in our output file. 1. Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 1231 Done with nonmem execution 1. Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 163 Done with nonmem execution Details: NONMEM application version – NM750 Slurm version - 21.08.6 MPICH version - 3.2.1 OS – Amazon Linux 2 Please let me know if you need anything from my end. Best Regards, Thomas ******************************************************************** This e-mail and any attachment hereto, is intended only for use by the addressee(s) named above and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, any dissemination, distribution or copying of this email, or any attachment hereto, is strictly prohibited. If you receive this email in error please immediately notify me by return electronic mail and permanently delete this email and any attachment hereto, any copy of this e-mail and of any such attachment, and any printout thereof. Finally, please note that only authorized representatives of Regeneron Pharmaceuticals, Inc. have the power and authority to enter into business dealings with any third party. ********************************************************************

1 0

Re: [mpich-discuss] Issue in MPICH while submitting jobs through slurm in NONMEM application
by Thomas Jayaseelan-External 15 Jun '23

15 Jun '23

Hi All, It would helpful if you could help me on the below issue that we face in our application using MPI. Best Regards, Thomas From: Thomas Jayaseelan-External Sent: Thursday, June 15, 2023 10:58 AM To: discuss(a)mpich.org Cc: Sundaresh Krishnasamy-External <sundaresh.krishnasam(a)regeneron.com>; Hariram Jayaram-External <hariram.jayaram(a)regeneron.com> Subject: Issue in MPICH while submitting jobs through slurm in NONMEM application Hi Team, This is Thomas, I am part of HPCOPs team in Regeneron Pharmaceuticals company. We build and support the HPC cluster infrastructure for the business as per their requirements. I have reached out to you to get help on an issue that we are currently facing with MPI. It would be great if you could help us in getting a solution to it. Nonmem is the application in which users submit the jobs through CLI, it is a CLI based application. When user tries to run job with more no. of cores the job runs for 10 to 15 hours and then stops intermittently. Please find the below error message that we get in our output file. 1. Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 1231 Done with nonmem execution 1. Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 163 Done with nonmem execution Details: NONMEM application version - NM750 Slurm version - 21.08.6 MPICH version - 3.2.1 OS - Amazon Linux 2 Please let me know if you need anything from my end. Best Regards, Thomas ******************************************************************** This e-mail and any attachment hereto, is intended only for use by the addressee(s) named above and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, any dissemination, distribution or copying of this email, or any attachment hereto, is strictly prohibited. If you receive this email in error please immediately notify me by return electronic mail and permanently delete this email and any attachment hereto, any copy of this e-mail and of any such attachment, and any printout thereof. Finally, please note that only authorized representatives of Regeneron Pharmaceuticals, Inc. have the power and authority to enter into business dealings with any third party. ********************************************************************

1 0

Re: [mpich-discuss] Duplicate reception of messages
by Thakur, Rajeev 14 Jun '23

14 Jun '23

To help narrow down the problem, can you try your code with some other MPI implementation on any system? If it works, can you send us a small test program that reproduces the problem? Rajeev From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, June 14, 2023 at 4:13 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Duplicate reception of messages My code seems to be receiving particular messages more than once -- I can tell from the unique variables they contain. I have verified that each unique variable is being sent exactly once. The sender simply calls MPI_ISend to send a single message containing an array of variables. The receiver pseudo-code is as follows. Have I made a mistake in my use of MPI_Iprobe? MPI_Status status; int flag; // loop in case there was more than one message queued up due to asynchrony between // sender and receiver. Most of the time, there is only one message. do { flag = false; MPI_Iprobe (sender_rank, tag, intercom, &flag, &status); if (flag) { int len; MPI_Get_count (&status, data_type, &len); MPI_Recv (buffer, len, data_type, sender_rank, tag, intercom, MPI_STATUS_IGNORE); // at this point, sometimes I see duplicate variable values in the buffer, in different // calls to MPI_Recv. That is, s supposedly unique variable appears in two different // message. } } while(flag); Thanks for your help. Here is my build information: MPICH Version: 4.0.1 MPICH Release date: Tue Feb 22 16:37:51 CST 2022 MPICH Device: ch3:nemesis MPICH configure: --prefix=/opt/mpich --with-device=ch3:nemesis --with-slurm --disable-fortran -enable-debuginfo --enable-g=debug MPICH CC: gcc -g -O2 MPICH CXX: g++ -g -O2 MPICH F77: -g MPICH FC: -g

1 0

Duplicate reception of messages
by Mccall, Kurt E. (MSFC-EV41) 14 Jun '23

14 Jun '23

My code seems to be receiving particular messages more than once -- I can tell from the unique variables they contain. I have verified that each unique variable is being sent exactly once. The sender simply calls MPI_ISend to send a single message containing an array of variables. The receiver pseudo-code is as follows. Have I made a mistake in my use of MPI_Iprobe? MPI_Status status; int flag; // loop in case there was more than one message queued up due to asynchrony between // sender and receiver. Most of the time, there is only one message. do { flag = false; MPI_Iprobe (sender_rank, tag, intercom, &flag, &status); if (flag) { int len; MPI_Get_count (&status, data_type, &len); MPI_Recv (buffer, len, data_type, sender_rank, tag, intercom, MPI_STATUS_IGNORE); // at this point, sometimes I see duplicate variable values in the buffer, in different // calls to MPI_Recv. That is, s supposedly unique variable appears in two different // message. } } while(flag); Thanks for your help. Here is my build information: MPICH Version: 4.0.1 MPICH Release date: Tue Feb 22 16:37:51 CST 2022 MPICH Device: ch3:nemesis MPICH configure: --prefix=/opt/mpich --with-device=ch3:nemesis --with-slurm --disable-fortran -enable-debuginfo --enable-g=debug MPICH CC: gcc -g -O2 MPICH CXX: g++ -g -O2 MPICH F77: -g MPICH FC: -g

1 0

Re: [mpich-discuss] Ask for help with MPICH
by Zhou, Hui 14 Jun '23

14 Jun '23

Hi Xuwen, The release mpich-4.1.2 contains hydra-4.1.2. There is no difference. The hydra release is for those users who only need the job launcher. --- Hui Zhou ________________________________ From: Xuwen XW3 Dai via discuss <discuss(a)mpich.org> Sent: Monday, June 12, 2023 8:59 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Xuwen XW3 Dai <daixw3(a)lenovo.com> Subject: [mpich-discuss] Ask for help with MPICH ********************************** IMPORTANT NOTICE: This email is from a third-party vendor and its personnel who provide services to Lenovo,and this email is not legally binding on Lenovo. Any Lenovo commitments should only be included in a formal contract signed between the two parties. Lenovo shall not be liable for any damages of any kind in any form whatsoever caused by this email. Any opinions expressed by individuals in this email neither represent nor reflect Lenovo's. This email and any documents sent with it may contain confidential information and are protected by the privileges of lawyers and clients. This message is only read and used by the intended recipients. Unauthorized viewing, use, disclosure or distribution of this email or the documents sent with it is strictly prohibited. If you are not the intended recipient, please do not read or save this message, and please delete this message permanently and destroy any printed copies. Thank you. 重要通知：本电子邮件来自于向联想提供服务的第三方供应商及其人员，且该邮件并不对联想发生法律约束力。联想的任何承诺均应仅包含在双方之间相互签署的正式合同中。对于本邮件无论基于何种原因导致的无论何种形式的任何损失，联想均不承担责任。个人在本邮件中表达的任何观点既不代表也不反映联想的观点。本邮件和与之一起发送的任何文件均可能含有保密信息，并受律师和客户特权保护。本邮件仅供指定的收件人阅读和使用。严禁对本邮件或与之一起发送的文件进行未经授权的查看、使用、披露或分发。如果您不是指定的收件人，请勿阅读或保存本邮件，并请永久删除本邮件并销毁任何打印副本。谢谢。 ********************************** Hi, developer of MPICH Hello, My name is Dai Xuwen, a developer of Lenovo. Recently, we are developing a project on a supercomputing platform, and we need to use the PBS scheduler to run the linpack job, and the mpi used is MPICH. I would like to ask what is the difference between mpich-4.1.2 (stable release) and hydra-4.1.2 (stable release) in the download options provided by the project official website? The linpack job I compiled using MPICH-4.1.2 can run, but PBS cannot obtain all the job process information of the multi-machine nodes. However, I use hydra-4.1.2 to run the linpack multi-machine job, and pbs can get the process information of all nodes. I can't understand the difference between the two installation packages hydra-4.1.2 and mpich-4.1.2 from the documents you provided, and I hope to get your help, thank you. This email uses Google translation, I hope you can understand it. Hi，MPICH的开发者你好，我是戴旭文，一名联想公司的开发人员。最近我们在开发一个超算平台的项目，需要使用PBS调度器来运行linpack作业，使用的mpi是MPICH。我想请问一下项目官网提供的下载选项里面，mpich-4.1.2 (stable release)和hydra-4.1.2 (stable release)有什么区别？我使用MPICH-4.1.2编译的linpack作业可以运行，但是PBS获取不到多机节点的所有作业进程信息。但是，我使用hydra-4.1.2 来运行的linpack多机作业，pbs就可以获取所有节点的进程信息。我从你们提供的文档里无法理解hydra-4.1.2 和mpich-4.1.2这两个安装包的区别，希望能得到您的帮助，谢谢。这封邮件使用谷歌翻译，希望您能看懂 BRs Dai Xuwen

1 0