- discuss - mailman3.cels.anl.gov

Ask for help with MPICH
by Xuwen XW3 Dai 12 Jun '23

12 Jun '23

********************************** IMPORTANT NOTICE: This email is from a third-party vendor and its personnel who provide services to Lenovo,and this email is not legally binding on Lenovo. Any Lenovo commitments should only be included in a formal contract signed between the two parties. Lenovo shall not be liable for any damages of any kind in any form whatsoever caused by this email. Any opinions expressed by individuals in this email neither represent nor reflect Lenovo's. This email and any documents sent with it may contain confidential information and are protected by the privileges of lawyers and clients. This message is only read and used by the intended recipients. Unauthorized viewing, use, disclosure or distribution of this email or the documents sent with it is strictly prohibited. If you are not the intended recipient, please do not read or save this message, and please delete this message permanently and destroy any printed copies. Thank you. 重要通知：本电子邮件来自于向联想提供服务的第三方供应商及其人员，且该邮件并不对联想发生法律约束力。联想的任何承诺均应仅包含在双方之间相互签署的正式合同中。对于本邮件无论基于何种原因导致的无论何种形式的任何损失，联想均不承担责任。个人在本邮件中表达的任何观点既不代表也不反映联想的观点。本邮件和与之一起发送的任何文件均可能含有保密信息，并受律师和客户特权保护。本邮件仅供指定的收件人阅读和使用。严禁对本邮件或与之一起发送的文件进行未经授权的查看、使用、披露或分发。如果您不是指定的收件人，请勿阅读或保存本邮件，并请永久删除本邮件并销毁任何打印副本。谢谢。 ********************************** Hi, developer of MPICH Hello, My name is Dai Xuwen, a developer of Lenovo. Recently, we are developing a project on a supercomputing platform, and we need to use the PBS scheduler to run the linpack job, and the mpi used is MPICH. I would like to ask what is the difference between mpich-4.1.2 (stable release) and hydra-4.1.2 (stable release) in the download options provided by the project official website? The linpack job I compiled using MPICH-4.1.2 can run, but PBS cannot obtain all the job process information of the multi-machine nodes. However, I use hydra-4.1.2 to run the linpack multi-machine job, and pbs can get the process information of all nodes. I can't understand the difference between the two installation packages hydra-4.1.2 and mpich-4.1.2 from the documents you provided, and I hope to get your help, thank you. This email uses Google translation, I hope you can understand it. Hi，MPICH的开发者你好，我是戴旭文，一名联想公司的开发人员。最近我们在开发一个超算平台的项目，需要使用PBS调度器来运行linpack作业，使用的mpi是MPICH。我想请问一下项目官网提供的下载选项里面，mpich-4.1.2 (stable release)和hydra-4.1.2 (stable release)有什么区别？我使用MPICH-4.1.2编译的linpack作业可以运行，但是PBS获取不到多机节点的所有作业进程信息。但是，我使用hydra-4.1.2 来运行的linpack多机作业，pbs就可以获取所有节点的进程信息。我从你们提供的文档里无法理解hydra-4.1.2 和mpich-4.1.2这两个安装包的区别，希望能得到您的帮助，谢谢。这封邮件使用谷歌翻译，希望您能看懂 BRs Dai Xuwen

1 0

Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after
by Solomonik, Edgar 08 Jun '23

08 Jun '23

Thanks, indeed the env variable fixes the issue and glad to hear its fixed in the latest version. Best, Edgar ________________________________ From: Raffenetti, Ken via discuss <discuss(a)mpich.org> Sent: Thursday, June 8, 2023 3:50 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Raffenetti, Ken <raffenet(a)anl.gov> Subject: Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after FWIW, you can workaround the bug in older versions by setting MPIR_CVAR_DEVICE_COLLECTIVES=none in your environment. Ken From: "Raffenetti, Ken via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, June 8, 2023 at 3:45 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Raffenetti, Ken" <raffenet(a)anl.gov> Subject: Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after Hi, I believe this bug was recently fixed in https://github.com/pmodels/mpich/pull/6543<https://urldefense.com/v3/__https://github.com/pmodels/mpich/pull/6543__;!!…>. The fix is part of the MPICH 4.1.2 release just posted to our website and Github. I confirmed that your test program works as expected now vs. an older 4.1 release. Ken From: "Solomonik, Edgar via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, June 8, 2023 at 3:37 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Solomonik, Edgar" <solomon2(a)illinois.edu> Subject: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after Hello, Our library's autobuild (CTF, which uses MPI extensively and in relatively sophisticated ways) started failing on multiple architectures after github workflows moved to later OS versions (and so later MPI versions). I believe I have narrowed the issue to an MPI bug associated with very basic usage of MPI Reduce. The following test code runs into a segmentation fault inside MPI when running with 2 MPI processes with the latest Ubuntu MPI build and MPI 4.0. It works for smaller values of message size (n) or if the root is rank 0. The usage of MPI_IN_PLACE adheres with the MPI standard. Best, Edgar Solomonik #include <mpi.h> #include <iostream> int main(int argc, char ** argv){ int64_t n = 257; MPI_Init(&argc, &argv); int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); double * A = (double*)malloc(sizeof(double)*n); for (int i=0; i<n; i++){ A[i] = (double)i; } if (rank == 1){ MPI_Reduce(MPI_IN_PLACE, A, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } else { MPI_Reduce(A, NULL, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } free(A); MPI_Finalize(); return 0; }

1 0

Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after
by Raffenetti, Ken 08 Jun '23

08 Jun '23

FWIW, you can workaround the bug in older versions by setting MPIR_CVAR_DEVICE_COLLECTIVES=none in your environment. Ken From: "Raffenetti, Ken via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, June 8, 2023 at 3:45 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Raffenetti, Ken" <raffenet(a)anl.gov> Subject: Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after Hi, I believe this bug was recently fixed in https://github.com/pmodels/mpich/pull/6543. The fix is part of the MPICH 4.1.2 release just posted to our website and Github. I confirmed that your test program works as expected now vs. an older 4.1 release. Ken From: "Solomonik, Edgar via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, June 8, 2023 at 3:37 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Solomonik, Edgar" <solomon2(a)illinois.edu> Subject: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after Hello, Our library's autobuild (CTF, which uses MPI extensively and in relatively sophisticated ways) started failing on multiple architectures after github workflows moved to later OS versions (and so later MPI versions). I believe I have narrowed the issue to an MPI bug associated with very basic usage of MPI Reduce. The following test code runs into a segmentation fault inside MPI when running with 2 MPI processes with the latest Ubuntu MPI build and MPI 4.0. It works for smaller values of message size (n) or if the root is rank 0. The usage of MPI_IN_PLACE adheres with the MPI standard. Best, Edgar Solomonik #include <mpi.h> #include <iostream> int main(int argc, char ** argv){ int64_t n = 257; MPI_Init(&argc, &argv); int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); double * A = (double*)malloc(sizeof(double)*n); for (int i=0; i<n; i++){ A[i] = (double)i; } if (rank == 1){ MPI_Reduce(MPI_IN_PLACE, A, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } else { MPI_Reduce(A, NULL, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } free(A); MPI_Finalize(); return 0; }

1 0

Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after
by Raffenetti, Ken 08 Jun '23

08 Jun '23

Hi, I believe this bug was recently fixed in https://github.com/pmodels/mpich/pull/6543. The fix is part of the MPICH 4.1.2 release just posted to our website and Github. I confirmed that your test program works as expected now vs. an older 4.1 release. Ken From: "Solomonik, Edgar via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, June 8, 2023 at 3:37 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Solomonik, Edgar" <solomon2(a)illinois.edu> Subject: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after Hello, Our library's autobuild (CTF, which uses MPI extensively and in relatively sophisticated ways) started failing on multiple architectures after github workflows moved to later OS versions (and so later MPI versions). I believe I have narrowed the issue to an MPI bug associated with very basic usage of MPI Reduce. The following test code runs into a segmentation fault inside MPI when running with 2 MPI processes with the latest Ubuntu MPI build and MPI 4.0. It works for smaller values of message size (n) or if the root is rank 0. The usage of MPI_IN_PLACE adheres with the MPI standard. Best, Edgar Solomonik #include <mpi.h> #include <iostream> int main(int argc, char ** argv){ int64_t n = 257; MPI_Init(&argc, &argv); int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); double * A = (double*)malloc(sizeof(double)*n); for (int i=0; i<n; i++){ A[i] = (double)i; } if (rank == 1){ MPI_Reduce(MPI_IN_PLACE, A, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } else { MPI_Reduce(A, NULL, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } free(A); MPI_Finalize(); return 0; }

1 0

MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after
by Solomonik, Edgar 08 Jun '23

08 Jun '23

Hello, Our library's autobuild (CTF, which uses MPI extensively and in relatively sophisticated ways) started failing on multiple architectures after github workflows moved to later OS versions (and so later MPI versions). I believe I have narrowed the issue to an MPI bug associated with very basic usage of MPI Reduce. The following test code runs into a segmentation fault inside MPI when running with 2 MPI processes with the latest Ubuntu MPI build and MPI 4.0. It works for smaller values of message size (n) or if the root is rank 0. The usage of MPI_IN_PLACE adheres with the MPI standard. Best, Edgar Solomonik #include <mpi.h> #include <iostream> int main(int argc, char ** argv){ int64_t n = 257; MPI_Init(&argc, &argv); int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); double * A = (double*)malloc(sizeof(double)*n); for (int i=0; i<n; i++){ A[i] = (double)i; } if (rank == 1){ MPI_Reduce(MPI_IN_PLACE, A, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } else { MPI_Reduce(A, NULL, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } free(A); MPI_Finalize(); return 0; }

1 0

Re: [mpich-discuss] Building 4.1.1
by Raffenetti, Ken 31 May '23

31 May '23

Try adding –disable-opx to the configure line to disable the OPX provider in the embedded libfabric. There must be some incompatibility with the compiler that is not detected by the configure script. Ken From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, May 31, 2023 at 6:47 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Building 4.1.1 I’m trying to build 4.1.1 on our local cluster. I’m configuring with ./configure --prefix=/people/d3g293/mpich/mpich-4.1.1/install --with-device=ch4:ofi --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ (I also tried --with-device=ch4:ofi:sockets) and my environment is Currently Loaded Modulefiles: 1) gcc/8.1.0 3) cmake/3.22.1 5) mkl/2019u5 2) python/3.7.2 4) git/2.11.0(default) 6) texlive/2017 I keep getting errors that look like In file included from ./prov/opx/include/rdma/opx/fi_opx_reliability.h:36, from ./prov/opx/include/rdma/opx/fi_opx_domain.h:44, from prov/opx/src/fi_opx_cntr.c:35: ./prov/opx/include/rdma/opx/fi_opx_hfi1_transport.h: At top level: ./prov/opx/include/rdma/opx/fi_opx.h:209:44: error: expected identifier or ‘(’ before ‘if’ #define OPX_COMPILE_TIME_ASSERT(cond, msg) if(0){switch(0){case 0:case cond:;}} Any suggestions on how to cure this? I’ve had pretty good luck building earlier versions of MPICH (4.0.2) using this formula. Bruce Palmer Computer Scientist Pacific Northwest National Laboratory (509) 375-3899

1 0

Building 4.1.1
by Palmer, Bruce J 31 May '23

31 May '23

I'm trying to build 4.1.1 on our local cluster. I'm configuring with ./configure --prefix=/people/d3g293/mpich/mpich-4.1.1/install --with-device=ch4:ofi --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ (I also tried --with-device=ch4:ofi:sockets) and my environment is Currently Loaded Modulefiles: 1) gcc/8.1.0 3) cmake/3.22.1 5) mkl/2019u5 2) python/3.7.2 4) git/2.11.0(default) 6) texlive/2017 I keep getting errors that look like In file included from ./prov/opx/include/rdma/opx/fi_opx_reliability.h:36, from ./prov/opx/include/rdma/opx/fi_opx_domain.h:44, from prov/opx/src/fi_opx_cntr.c:35: ./prov/opx/include/rdma/opx/fi_opx_hfi1_transport.h: At top level: ./prov/opx/include/rdma/opx/fi_opx.h:209:44: error: expected identifier or '(' before 'if' #define OPX_COMPILE_TIME_ASSERT(cond, msg) if(0){switch(0){case 0:case cond:;}} Any suggestions on how to cure this? I've had pretty good luck building earlier versions of MPICH (4.0.2) using this formula. Bruce Palmer Computer Scientist Pacific Northwest National Laboratory (509) 375-3899

1 0

[REMINDER] Join the MPICH team at ECP Community BoF Days today!
by Raffenetti, Ken 15 Feb '23

15 Feb '23

All, The MPICH team will host a session today as part of the ECP Community BoF Days (https://www.exascaleproject.org/event/2023-ecp-community-bof-days/) We will discuss MPICH release updates as well as hear from some of our partners. The event is virtual and registration is free at this link: https://uoregon.zoom.us/meeting/register/tJMlde-gqjItEte5PxaSSXa9iO275T9SRu… Date and Time: February 15th, 2:00pm - 3:30pm Central Agenda: Intro - Ken Raffenetti (ANL) MPICH Update - Yanfei Guo (ANL) Partner Updates (10 minutes each) - MVAPICH2 Update - Hari Subramoni (OSU) - Intel Update - Gengbin Zheng - NVIDIA Update - Jim Dinan - ParTec Update - Simon Pickartz - HPE/Cray - Krishna Kandalla Q&A This information is also available on our wiki: https://github.com/pmodels/mpich/wiki/2023-ECP-Community-BoF Cheers, The MPICH team

1 0

Join the MPICH team at ECP Community BoF Days next week!
by Raffenetti, Ken 09 Feb '23

09 Feb '23

All, The MPICH team will host a session as part of next week's ECP Community BoF Days (https://www.exascaleproject.org/event/2023-ecp-community-bof-days/) We will discuss MPICH release updates as well as hear from some of our partners. The event is virtual and registration is free at this link: https://uoregon.zoom.us/meeting/register/tJMlde-gqjItEte5PxaSSXa9iO275T9SRu… Date and Time: February 15th, 2:00pm - 3:30pm Central Agenda (subject to change): Intro - Ken Raffenetti (ANL) MPICH Update - Yanfei Guo (ANL) Partner Updates (10 minutes each) - MVAPICH2 Update - Hari Subramoni (OSU) - Intel Update - Gengbin Zheng - NVIDIA Update - Jim Dinan - ParTec Update - TBD Q&A This information is also available on our wiki: https://github.com/pmodels/mpich/wiki/2023-ECP-Community-BoF Cheers, The MPICH team

1 0

Re: [mpich-discuss] MPICH Test Suite - limit number of processes
by Max R. Dechantsreiter 16 Jan '23

16 Jan '23

Hi Joachim, Thank you, I will try that overnight. This setting might be useful in practical situations. Interesting that some of the tests require 33 processes: that is high even for most modern processors, so I presume the idea was to run the suite on multiple nodes. Best regards, Max --- On Sun, Jan 15, 2023 at 09:19:58PM +0000, Jenke, Joachim wrote: > For our unit testing (not mpich specific) I found it useful to set the following environmental variable to reduce the time wasted in busy waiting while we run our tests massively oversubscribed: > export MPIR_CVAR_POLLS_BEFORE_YIELD=1 > > - Joachim > ________________________________ > From: Max R. Dechantsreiter via discuss <discuss(a)mpich.org> > Sent: Sunday, January 15, 2023 3:09:19 PM > To: Zhou, Hui <zhouh(a)anl.gov> > Cc: Max R. Dechantsreiter <max(a)performancejones.com>; discuss(a)mpich.org <discuss(a)mpich.org> > Subject: Re: [mpich-discuss] MPICH Test Suite - limit number of processes > > Thank you for your suggestion. > > However, I did not see how to carry it through; perhaps one > day I will comb through the User's Guide. > > Instead I ran 'make testing' to completion on four cores: > elapsed time was 2h36m53s on Intel(R) Xeon(R) CPU E5-2620 v4 > (2.10GHz); all but two tests passed, with the two that did > not pass simply timing out (after 180s). > > Then I ran it again after modifying "runtests" thus: > > sed -i 's/^$ppnMax = -1;/$ppnMax = 4;/' $RUNTESTS > sed -i 's/^$np_max = -1;/$np_max = 4;/' $RUNTESTS > > where RUNTESTS=~test/mpi/runtests. (Alternatively these > variables could be modified in $RUNTESTS.in before prior to > building MPICH.) > > Changing those variables limited the maximum number of > processes to 4 as desired. The salient issues were the > warnings 'Number of processors must divide 10' that I > presume could be avoided by setting $np_max=2 or 5; the > number of tests affected was relatively small: 13 of 2524 > tests failed, with no test timing out. Elapsed time was > 1h04m28s - so the savings in time was significant but not > spectacular, suggesting that relatively few tests in the > original run oversubscribed the cores available in my VPS. > > > On Thu, Jan 12, 2023 at 04:29:07PM +0000, Zhou, Hui wrote: > > Unfortunately, the number of processes for each test is hardcoded in the testlist files. There is no mechanism for filtering or dynamically modifying them. You could either maintain a custom patch to modify the testlist, or have a script applying a separate xfail list. An example of such xfail list that we used in our internal testing is in test/mpi/maint/jenkins/xfail.conf. If the custom patch solution won't work in your situation, please open an github issue describing your use case, and we can discuss to add a filtering option to the runtests script. > > > > -- > > Hui > > ________________________________ > > From: Max R. Dechantsreiter via discuss <discuss(a)mpich.org> > > Sent: Thursday, January 12, 2023 9:58 AM > > To: discuss(a)mpich.org <discuss(a)mpich.org> > > Cc: Max R. Dechantsreiter <max(a)performancejones.com> > > Subject: [mpich-discuss] MPICH Test Suite - limit number of processes > > > > I would like to limit the number of processes used by the > > MPICH Test Suite, which I run by "make testing" after > > building MPICH. > > > > I tried > > > > export MPITEST_PPNMAX=<n> # The maximum number of processes per node. > > > > but this had no apparent effect: some tests still ran > > using more than 4 processes, when I replaced <n> by 4. > > How could I restrict these tests to run with maximum 4 > > processes, for example? > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0