June 2023 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_Init fail
by Raffenetti, Ken 26 Jun '23

26 Jun '23

The looks to be an error from a fairly old version of MPICH. Have you tried installing the latest version (4.1.2) available on our website or Github? Ken From: Nazmul Haque via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Sunday, June 25, 2023 at 11:38 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Nazmul Haque <shadhin873(a)outlook.com> Subject: [mpich-discuss] MPI_Init fail Hello, I have installed mpich2 and tried to run an executable with 'mpiexec -n 8 lmp -in .....' but it shows error, replying that mpi_init is unable to connect to the local host. How can I solve this? I have also attached the image of the error. Thank you for your time. Regards, Nazmul

1 0

MPI_Init fail
by Nazmul Haque 25 Jun '23

25 Jun '23

Hello, I have installed mpich2 and tried to run an executable with 'mpiexec -n 8 lmp -in .....' but it shows error, replying that mpi_init is unable to connect to the local host. How can I solve this? I have also attached the image of the error. Thank you for your time. Regards, Nazmul

1 0

Re: [mpich-discuss] Issue in MPICH while submitting jobs through slurm in NONMEM application
by Zhou, Hui 15 Jun '23

15 Jun '23

Hi Thomas, The assertion error means packet corruption. It's not clear what could be the cause unless you can provide a reproducer. Anyway, mpich-3.2.1 is quite old. My first suggestion would be try a newer mpich see if the error persists. -- Hui Zhou ________________________________ From: Thomas Jayaseelan-External via discuss <discuss(a)mpich.org> Sent: Thursday, June 15, 2023 9:58 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thomas Jayaseelan-External <thomas.jayaseelan(a)regeneron.com>; Sundaresh Krishnasamy-External <sundaresh.krishnasam(a)regeneron.com>; Hariram Jayaram-External <hariram.jayaram(a)regeneron.com> Subject: Re: [mpich-discuss] Issue in MPICH while submitting jobs through slurm in NONMEM application Hi All, It would helpful if you could help me on the below issue that we face in our application using MPI. Best Regards, Thomas From: Thomas Jayaseelan-External Sent: Thursday, June 15, 2023 10:58 AM To: discuss(a)mpich.org Cc: Sundaresh Krishnasamy-External <sundaresh.krishnasam(a)regeneron.com>; Hariram Jayaram-External <hariram.jayaram(a)regeneron.com> Subject: Issue in MPICH while submitting jobs through slurm in NONMEM application Hi Team, This is Thomas, I am part of HPCOPs team in Regeneron Pharmaceuticals company. We build and support the HPC cluster infrastructure for the business as per their requirements. I have reached out to you to get help on an issue that we are currently facing with MPI. It would be great if you could help us in getting a solution to it. Nonmem is the application in which users submit the jobs through CLI, it is a CLI based application. When user tries to run job with more no. of cores the job runs for 10 to 15 hours and then stops intermittently. Please find the below error message that we get in our output file. 1. Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 1231 Done with nonmem execution 1. Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 163 Done with nonmem execution Details: NONMEM application version – NM750 Slurm version - 21.08.6 MPICH version - 3.2.1 OS – Amazon Linux 2 Please let me know if you need anything from my end. Best Regards, Thomas ******************************************************************** This e-mail and any attachment hereto, is intended only for use by the addressee(s) named above and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, any dissemination, distribution or copying of this email, or any attachment hereto, is strictly prohibited. If you receive this email in error please immediately notify me by return electronic mail and permanently delete this email and any attachment hereto, any copy of this e-mail and of any such attachment, and any printout thereof. Finally, please note that only authorized representatives of Regeneron Pharmaceuticals, Inc. have the power and authority to enter into business dealings with any third party. ********************************************************************

1 0

Re: [mpich-discuss] Issue in MPICH while submitting jobs through slurm in NONMEM application
by Thomas Jayaseelan-External 15 Jun '23

15 Jun '23

Hi All, It would helpful if you could help me on the below issue that we face in our application using MPI. Best Regards, Thomas From: Thomas Jayaseelan-External Sent: Thursday, June 15, 2023 10:58 AM To: discuss(a)mpich.org Cc: Sundaresh Krishnasamy-External <sundaresh.krishnasam(a)regeneron.com>; Hariram Jayaram-External <hariram.jayaram(a)regeneron.com> Subject: Issue in MPICH while submitting jobs through slurm in NONMEM application Hi Team, This is Thomas, I am part of HPCOPs team in Regeneron Pharmaceuticals company. We build and support the HPC cluster infrastructure for the business as per their requirements. I have reached out to you to get help on an issue that we are currently facing with MPI. It would be great if you could help us in getting a solution to it. Nonmem is the application in which users submit the jobs through CLI, it is a CLI based application. When user tries to run job with more no. of cores the job runs for 10 to 15 hours and then stops intermittently. Please find the below error message that we get in our output file. 1. Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 1231 Done with nonmem execution 1. Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 163 Done with nonmem execution Details: NONMEM application version - NM750 Slurm version - 21.08.6 MPICH version - 3.2.1 OS - Amazon Linux 2 Please let me know if you need anything from my end. Best Regards, Thomas ******************************************************************** This e-mail and any attachment hereto, is intended only for use by the addressee(s) named above and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, any dissemination, distribution or copying of this email, or any attachment hereto, is strictly prohibited. If you receive this email in error please immediately notify me by return electronic mail and permanently delete this email and any attachment hereto, any copy of this e-mail and of any such attachment, and any printout thereof. Finally, please note that only authorized representatives of Regeneron Pharmaceuticals, Inc. have the power and authority to enter into business dealings with any third party. ********************************************************************

1 0

Re: [mpich-discuss] Duplicate reception of messages
by Thakur, Rajeev 14 Jun '23

14 Jun '23

To help narrow down the problem, can you try your code with some other MPI implementation on any system? If it works, can you send us a small test program that reproduces the problem? Rajeev From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, June 14, 2023 at 4:13 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Duplicate reception of messages My code seems to be receiving particular messages more than once -- I can tell from the unique variables they contain. I have verified that each unique variable is being sent exactly once. The sender simply calls MPI_ISend to send a single message containing an array of variables. The receiver pseudo-code is as follows. Have I made a mistake in my use of MPI_Iprobe? MPI_Status status; int flag; // loop in case there was more than one message queued up due to asynchrony between // sender and receiver. Most of the time, there is only one message. do { flag = false; MPI_Iprobe (sender_rank, tag, intercom, &flag, &status); if (flag) { int len; MPI_Get_count (&status, data_type, &len); MPI_Recv (buffer, len, data_type, sender_rank, tag, intercom, MPI_STATUS_IGNORE); // at this point, sometimes I see duplicate variable values in the buffer, in different // calls to MPI_Recv. That is, s supposedly unique variable appears in two different // message. } } while(flag); Thanks for your help. Here is my build information: MPICH Version: 4.0.1 MPICH Release date: Tue Feb 22 16:37:51 CST 2022 MPICH Device: ch3:nemesis MPICH configure: --prefix=/opt/mpich --with-device=ch3:nemesis --with-slurm --disable-fortran -enable-debuginfo --enable-g=debug MPICH CC: gcc -g -O2 MPICH CXX: g++ -g -O2 MPICH F77: -g MPICH FC: -g

1 0

Duplicate reception of messages
by Mccall, Kurt E. (MSFC-EV41) 14 Jun '23

14 Jun '23

My code seems to be receiving particular messages more than once -- I can tell from the unique variables they contain. I have verified that each unique variable is being sent exactly once. The sender simply calls MPI_ISend to send a single message containing an array of variables. The receiver pseudo-code is as follows. Have I made a mistake in my use of MPI_Iprobe? MPI_Status status; int flag; // loop in case there was more than one message queued up due to asynchrony between // sender and receiver. Most of the time, there is only one message. do { flag = false; MPI_Iprobe (sender_rank, tag, intercom, &flag, &status); if (flag) { int len; MPI_Get_count (&status, data_type, &len); MPI_Recv (buffer, len, data_type, sender_rank, tag, intercom, MPI_STATUS_IGNORE); // at this point, sometimes I see duplicate variable values in the buffer, in different // calls to MPI_Recv. That is, s supposedly unique variable appears in two different // message. } } while(flag); Thanks for your help. Here is my build information: MPICH Version: 4.0.1 MPICH Release date: Tue Feb 22 16:37:51 CST 2022 MPICH Device: ch3:nemesis MPICH configure: --prefix=/opt/mpich --with-device=ch3:nemesis --with-slurm --disable-fortran -enable-debuginfo --enable-g=debug MPICH CC: gcc -g -O2 MPICH CXX: g++ -g -O2 MPICH F77: -g MPICH FC: -g

1 0

Re: [mpich-discuss] Ask for help with MPICH
by Zhou, Hui 14 Jun '23

14 Jun '23

Hi Xuwen, The release mpich-4.1.2 contains hydra-4.1.2. There is no difference. The hydra release is for those users who only need the job launcher. --- Hui Zhou ________________________________ From: Xuwen XW3 Dai via discuss <discuss(a)mpich.org> Sent: Monday, June 12, 2023 8:59 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Xuwen XW3 Dai <daixw3(a)lenovo.com> Subject: [mpich-discuss] Ask for help with MPICH ********************************** IMPORTANT NOTICE: This email is from a third-party vendor and its personnel who provide services to Lenovo,and this email is not legally binding on Lenovo. Any Lenovo commitments should only be included in a formal contract signed between the two parties. Lenovo shall not be liable for any damages of any kind in any form whatsoever caused by this email. Any opinions expressed by individuals in this email neither represent nor reflect Lenovo's. This email and any documents sent with it may contain confidential information and are protected by the privileges of lawyers and clients. This message is only read and used by the intended recipients. Unauthorized viewing, use, disclosure or distribution of this email or the documents sent with it is strictly prohibited. If you are not the intended recipient, please do not read or save this message, and please delete this message permanently and destroy any printed copies. Thank you. 重要通知：本电子邮件来自于向联想提供服务的第三方供应商及其人员，且该邮件并不对联想发生法律约束力。联想的任何承诺均应仅包含在双方之间相互签署的正式合同中。对于本邮件无论基于何种原因导致的无论何种形式的任何损失，联想均不承担责任。个人在本邮件中表达的任何观点既不代表也不反映联想的观点。本邮件和与之一起发送的任何文件均可能含有保密信息，并受律师和客户特权保护。本邮件仅供指定的收件人阅读和使用。严禁对本邮件或与之一起发送的文件进行未经授权的查看、使用、披露或分发。如果您不是指定的收件人，请勿阅读或保存本邮件，并请永久删除本邮件并销毁任何打印副本。谢谢。 ********************************** Hi, developer of MPICH Hello, My name is Dai Xuwen, a developer of Lenovo. Recently, we are developing a project on a supercomputing platform, and we need to use the PBS scheduler to run the linpack job, and the mpi used is MPICH. I would like to ask what is the difference between mpich-4.1.2 (stable release) and hydra-4.1.2 (stable release) in the download options provided by the project official website? The linpack job I compiled using MPICH-4.1.2 can run, but PBS cannot obtain all the job process information of the multi-machine nodes. However, I use hydra-4.1.2 to run the linpack multi-machine job, and pbs can get the process information of all nodes. I can't understand the difference between the two installation packages hydra-4.1.2 and mpich-4.1.2 from the documents you provided, and I hope to get your help, thank you. This email uses Google translation, I hope you can understand it. Hi，MPICH的开发者你好，我是戴旭文，一名联想公司的开发人员。最近我们在开发一个超算平台的项目，需要使用PBS调度器来运行linpack作业，使用的mpi是MPICH。我想请问一下项目官网提供的下载选项里面，mpich-4.1.2 (stable release)和hydra-4.1.2 (stable release)有什么区别？我使用MPICH-4.1.2编译的linpack作业可以运行，但是PBS获取不到多机节点的所有作业进程信息。但是，我使用hydra-4.1.2 来运行的linpack多机作业，pbs就可以获取所有节点的进程信息。我从你们提供的文档里无法理解hydra-4.1.2 和mpich-4.1.2这两个安装包的区别，希望能得到您的帮助，谢谢。这封邮件使用谷歌翻译，希望您能看懂 BRs Dai Xuwen

1 0

Ask for help with MPICH
by Xuwen XW3 Dai 12 Jun '23

12 Jun '23

********************************** IMPORTANT NOTICE: This email is from a third-party vendor and its personnel who provide services to Lenovo,and this email is not legally binding on Lenovo. Any Lenovo commitments should only be included in a formal contract signed between the two parties. Lenovo shall not be liable for any damages of any kind in any form whatsoever caused by this email. Any opinions expressed by individuals in this email neither represent nor reflect Lenovo's. This email and any documents sent with it may contain confidential information and are protected by the privileges of lawyers and clients. This message is only read and used by the intended recipients. Unauthorized viewing, use, disclosure or distribution of this email or the documents sent with it is strictly prohibited. If you are not the intended recipient, please do not read or save this message, and please delete this message permanently and destroy any printed copies. Thank you. 重要通知：本电子邮件来自于向联想提供服务的第三方供应商及其人员，且该邮件并不对联想发生法律约束力。联想的任何承诺均应仅包含在双方之间相互签署的正式合同中。对于本邮件无论基于何种原因导致的无论何种形式的任何损失，联想均不承担责任。个人在本邮件中表达的任何观点既不代表也不反映联想的观点。本邮件和与之一起发送的任何文件均可能含有保密信息，并受律师和客户特权保护。本邮件仅供指定的收件人阅读和使用。严禁对本邮件或与之一起发送的文件进行未经授权的查看、使用、披露或分发。如果您不是指定的收件人，请勿阅读或保存本邮件，并请永久删除本邮件并销毁任何打印副本。谢谢。 ********************************** Hi, developer of MPICH Hello, My name is Dai Xuwen, a developer of Lenovo. Recently, we are developing a project on a supercomputing platform, and we need to use the PBS scheduler to run the linpack job, and the mpi used is MPICH. I would like to ask what is the difference between mpich-4.1.2 (stable release) and hydra-4.1.2 (stable release) in the download options provided by the project official website? The linpack job I compiled using MPICH-4.1.2 can run, but PBS cannot obtain all the job process information of the multi-machine nodes. However, I use hydra-4.1.2 to run the linpack multi-machine job, and pbs can get the process information of all nodes. I can't understand the difference between the two installation packages hydra-4.1.2 and mpich-4.1.2 from the documents you provided, and I hope to get your help, thank you. This email uses Google translation, I hope you can understand it. Hi，MPICH的开发者你好，我是戴旭文，一名联想公司的开发人员。最近我们在开发一个超算平台的项目，需要使用PBS调度器来运行linpack作业，使用的mpi是MPICH。我想请问一下项目官网提供的下载选项里面，mpich-4.1.2 (stable release)和hydra-4.1.2 (stable release)有什么区别？我使用MPICH-4.1.2编译的linpack作业可以运行，但是PBS获取不到多机节点的所有作业进程信息。但是，我使用hydra-4.1.2 来运行的linpack多机作业，pbs就可以获取所有节点的进程信息。我从你们提供的文档里无法理解hydra-4.1.2 和mpich-4.1.2这两个安装包的区别，希望能得到您的帮助，谢谢。这封邮件使用谷歌翻译，希望您能看懂 BRs Dai Xuwen

1 0

Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after
by Solomonik, Edgar 08 Jun '23

08 Jun '23

Thanks, indeed the env variable fixes the issue and glad to hear its fixed in the latest version. Best, Edgar ________________________________ From: Raffenetti, Ken via discuss <discuss(a)mpich.org> Sent: Thursday, June 8, 2023 3:50 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Raffenetti, Ken <raffenet(a)anl.gov> Subject: Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after FWIW, you can workaround the bug in older versions by setting MPIR_CVAR_DEVICE_COLLECTIVES=none in your environment. Ken From: "Raffenetti, Ken via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, June 8, 2023 at 3:45 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Raffenetti, Ken" <raffenet(a)anl.gov> Subject: Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after Hi, I believe this bug was recently fixed in https://github.com/pmodels/mpich/pull/6543<https://urldefense.com/v3/__https://github.com/pmodels/mpich/pull/6543__;!!…>. The fix is part of the MPICH 4.1.2 release just posted to our website and Github. I confirmed that your test program works as expected now vs. an older 4.1 release. Ken From: "Solomonik, Edgar via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, June 8, 2023 at 3:37 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Solomonik, Edgar" <solomon2(a)illinois.edu> Subject: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after Hello, Our library's autobuild (CTF, which uses MPI extensively and in relatively sophisticated ways) started failing on multiple architectures after github workflows moved to later OS versions (and so later MPI versions). I believe I have narrowed the issue to an MPI bug associated with very basic usage of MPI Reduce. The following test code runs into a segmentation fault inside MPI when running with 2 MPI processes with the latest Ubuntu MPI build and MPI 4.0. It works for smaller values of message size (n) or if the root is rank 0. The usage of MPI_IN_PLACE adheres with the MPI standard. Best, Edgar Solomonik #include <mpi.h> #include <iostream> int main(int argc, char ** argv){ int64_t n = 257; MPI_Init(&argc, &argv); int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); double * A = (double*)malloc(sizeof(double)*n); for (int i=0; i<n; i++){ A[i] = (double)i; } if (rank == 1){ MPI_Reduce(MPI_IN_PLACE, A, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } else { MPI_Reduce(A, NULL, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } free(A); MPI_Finalize(); return 0; }

1 0

Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after
by Raffenetti, Ken 08 Jun '23

08 Jun '23

FWIW, you can workaround the bug in older versions by setting MPIR_CVAR_DEVICE_COLLECTIVES=none in your environment. Ken From: "Raffenetti, Ken via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, June 8, 2023 at 3:45 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Raffenetti, Ken" <raffenet(a)anl.gov> Subject: Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after Hi, I believe this bug was recently fixed in https://github.com/pmodels/mpich/pull/6543. The fix is part of the MPICH 4.1.2 release just posted to our website and Github. I confirmed that your test program works as expected now vs. an older 4.1 release. Ken From: "Solomonik, Edgar via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, June 8, 2023 at 3:37 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Solomonik, Edgar" <solomon2(a)illinois.edu> Subject: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after Hello, Our library's autobuild (CTF, which uses MPI extensively and in relatively sophisticated ways) started failing on multiple architectures after github workflows moved to later OS versions (and so later MPI versions). I believe I have narrowed the issue to an MPI bug associated with very basic usage of MPI Reduce. The following test code runs into a segmentation fault inside MPI when running with 2 MPI processes with the latest Ubuntu MPI build and MPI 4.0. It works for smaller values of message size (n) or if the root is rank 0. The usage of MPI_IN_PLACE adheres with the MPI standard. Best, Edgar Solomonik #include <mpi.h> #include <iostream> int main(int argc, char ** argv){ int64_t n = 257; MPI_Init(&argc, &argv); int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); double * A = (double*)malloc(sizeof(double)*n); for (int i=0; i<n; i++){ A[i] = (double)i; } if (rank == 1){ MPI_Reduce(MPI_IN_PLACE, A, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } else { MPI_Reduce(A, NULL, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } free(A); MPI_Finalize(); return 0; }

1 0