September 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Error Running MPICH for Photochemical Modeling
by Abhishek Bhat 12 Sep '14

12 Sep '14

Sangmin. I updated to mpich3 and getting the following error Fatal error in MPI_Recv: A process has failed, error stack: MPI_Recv(187).............: MPI_Recv(buf=0x7fff93840c30, count=644490, MPI_REAL, src=1, tag=14131, MPI_COMM_WORLD, status=0x7fff94444f20) failed dequeue_and_set_error(865): Communication error with rank 1 rank 1 in job 1 dfw-camx_55000 caused collective abort of all ranks exit status of rank 1: killed by signal 9 Same situation, successful runs for smaller resource runs and for up to 7 processes. Error at more than 7. Here is the mpich command I am using to run from my job file... cat << ieof > nodes dfw-camx:1 dfw-camx-n1:1 dfw-camx-n2:1 dfw-camx-n3:1 dfw-camx-n4:1 dfw-camx-n5:1 dfw-camx-n6:1 dfw-camx-n7:1 ieof set NUMPROCS = 8 set RING = `wc -l nodes | awk '{print $1}'` mpdboot -n $RING -f nodes -verbose if( ! { mpiexec -machinefile nodes -np $NUMPROCS $EXEC } ) then mpdallexit exit endif For a successful run the NUMPROCS has to be < = 7. Any help is much appreciated. Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Friday, September 12, 2014 1:11 PM To: <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Error Running MPICH for Photochemical Modeling Hi Abhishek, Can you try with the recent MPICH release to see if the same error happens? You can download the recent release, 3.1.2, from http://www.mpich.org/downloads/. Thanks, Sangmin On Sep 12, 2014, at 12:59 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: I am running a photochemical modeling on Linux cluster (CentOS_64 bit) with 1 master and 8 slave nodes with quad core (intel i7) on each node. I have two scenarios, in first scenario, I am running less data intensive run on all 8 nodes (NUMPROCS = 9) and the run will go fine. When running same configuration for a more intense run, I am getting following error. Fatal error in MPI_Recv: Other MPI error, error stack: MPI_Recv(187).....................: MPI_Recv(buf=0x7fff989d53b0, count=644490, MPI_REAL, src=1, tag=14131, MPI_COMM_WORLD, status=0x7fff995d96a0) failed MPIDI_CH3I_Progress(150)..........: MPID_nem_mpich2_blocking_recv(948): MPID_nem_tcp_connpoll(1720).......: state_commrdy_handler(1556).......: MPID_nem_tcp_recv_handler(1446)...: socket closed rank 1 in job 1 dfw-camx_55000 caused collective abort of all ranks exit status of rank 1: killed by signal 9 If I run the program with smaller nodes (smaller than 7 NUMPROCS) the run goes fine. It appears that the rank 1 (my first node) is collectively causing all the ranks, but I could identify why. I tried following solutions - 1. Increased master memory to 32 gb 2. Increased all nodes memory to 32 gb 3. Exchanged the rank 1 to different node in the parallel. In all situations, I am getting this error. Surprisingly, when I am running smaller (less data intensive runs), I am not getting this error even if I increase the NUMPROCS to 32 processes. Any help will be highly appreciated. I am running mpich 1.4 Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant Trinity Consultants 12770 Merit Drive, Suite 900 | Dallas, Texas 75251 Office: 972-661-8100| Mobile: 806-281-7617 Email: abhat(a)trinityconsultants.com<mailto:[email protected]> | LinkedIn: www.linkedin.com/in/abhattrinityconsultants<http://www.linkedin.com/in/abhattrinityconsultants> Stay current on environmental issues. Subscribe<http://www.trinityconsultants.com/Subscribe/> today to receive Trinity's free Environmental Quarterly<http://www.trinityconsultants.com/EnvironmentalQuarterly/>. Learn about Trinity's courses<http://www.trinityconsultants.com/Training/> for environmental professionals. <image001.gif><http://www.linkedin.com/company/trinity-consultants> <image002.gif><http://www.facebook.com/TrinityConsults> <image003.gif><http://twitter.com/trinityconsults> <image004.gif><http://www.youtube.com/trinityconsultants> <image005.jpg> _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________

1 0

Re: [mpich-discuss] Error Running MPICH for Photochemical Modeling
by Seo, Sangmin 12 Sep '14

12 Sep '14

Hi Abhishek, Can you try with the recent MPICH release to see if the same error happens? You can download the recent release, 3.1.2, from http://www.mpich.org/downloads/. Thanks, Sangmin On Sep 12, 2014, at 12:59 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: I am running a photochemical modeling on Linux cluster (CentOS_64 bit) with 1 master and 8 slave nodes with quad core (intel i7) on each node. I have two scenarios, in first scenario, I am running less data intensive run on all 8 nodes (NUMPROCS = 9) and the run will go fine. When running same configuration for a more intense run, I am getting following error. Fatal error in MPI_Recv: Other MPI error, error stack: MPI_Recv(187).....................: MPI_Recv(buf=0x7fff989d53b0, count=644490, MPI_REAL, src=1, tag=14131, MPI_COMM_WORLD, status=0x7fff995d96a0) failed MPIDI_CH3I_Progress(150)..........: MPID_nem_mpich2_blocking_recv(948): MPID_nem_tcp_connpoll(1720).......: state_commrdy_handler(1556).......: MPID_nem_tcp_recv_handler(1446)...: socket closed rank 1 in job 1 dfw-camx_55000 caused collective abort of all ranks exit status of rank 1: killed by signal 9 If I run the program with smaller nodes (smaller than 7 NUMPROCS) the run goes fine. It appears that the rank 1 (my first node) is collectively causing all the ranks, but I could identify why. I tried following solutions – 1. Increased master memory to 32 gb 2. Increased all nodes memory to 32 gb 3. Exchanged the rank 1 to different node in the parallel. In all situations, I am getting this error. Surprisingly, when I am running smaller (less data intensive runs), I am not getting this error even if I increase the NUMPROCS to 32 processes. Any help will be highly appreciated. I am running mpich 1.4 Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant Trinity Consultants 12770 Merit Drive, Suite 900 | Dallas, Texas 75251 Office: 972-661-8100| Mobile: 806-281-7617 Email: abhat(a)trinityconsultants.com<mailto:[email protected]> | LinkedIn: www.linkedin.com/in/abhattrinityconsultants<http://www.linkedin.com/in/abhattrinityconsultants> Stay current on environmental issues. Subscribe<http://www.trinityconsultants.com/Subscribe/> today to receive Trinity's free Environmental Quarterly<http://www.trinityconsultants.com/EnvironmentalQuarterly/>. Learn about Trinity’s courses<http://www.trinityconsultants.com/Training/> for environmental professionals. <image001.gif><http://www.linkedin.com/company/trinity-consultants> <image002.gif><http://www.facebook.com/TrinityConsults> <image003.gif><http://twitter.com/trinityconsults> <image004.gif><http://www.youtube.com/trinityconsultants> <image005.jpg> _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Error Running MPICH for Photochemical Modeling
by Abhishek Bhat 12 Sep '14

12 Sep '14

I am running a photochemical modeling on Linux cluster (CentOS_64 bit) with 1 master and 8 slave nodes with quad core (intel i7) on each node. I have two scenarios, in first scenario, I am running less data intensive run on all 8 nodes (NUMPROCS = 9) and the run will go fine. When running same configuration for a more intense run, I am getting following error. Fatal error in MPI_Recv: Other MPI error, error stack: MPI_Recv(187).....................: MPI_Recv(buf=0x7fff989d53b0, count=644490, MPI_REAL, src=1, tag=14131, MPI_COMM_WORLD, status=0x7fff995d96a0) failed MPIDI_CH3I_Progress(150)..........: MPID_nem_mpich2_blocking_recv(948): MPID_nem_tcp_connpoll(1720).......: state_commrdy_handler(1556).......: MPID_nem_tcp_recv_handler(1446)...: socket closed rank 1 in job 1 dfw-camx_55000 caused collective abort of all ranks exit status of rank 1: killed by signal 9 If I run the program with smaller nodes (smaller than 7 NUMPROCS) the run goes fine. It appears that the rank 1 (my first node) is collectively causing all the ranks, but I could identify why. I tried following solutions - 1. Increased master memory to 32 gb 2. Increased all nodes memory to 32 gb 3. Exchanged the rank 1 to different node in the parallel. In all situations, I am getting this error. Surprisingly, when I am running smaller (less data intensive runs), I am not getting this error even if I increase the NUMPROCS to 32 processes. Any help will be highly appreciated. I am running mpich 1.4 Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant Trinity Consultants 12770 Merit Drive, Suite 900 | Dallas, Texas 75251 Office: 972-661-8100| Mobile: 806-281-7617 Email: abhat(a)trinityconsultants.com<mailto:[email protected]> | LinkedIn: www.linkedin.com/in/abhattrinityconsultants<http://www.linkedin.com/in/abhattrinityconsultants> Stay current on environmental issues. Subscribe<http://www.trinityconsultants.com/Subscribe/> today to receive Trinity's free Environmental Quarterly<http://www.trinityconsultants.com/EnvironmentalQuarterly/>. Learn about Trinity's courses<http://www.trinityconsultants.com/Training/> for environmental professionals. [LinkedIn icon_23p]<http://www.linkedin.com/company/trinity-consultants> [Facebook icon_23p] <http://www.facebook.com/TrinityConsults> [Twitter icon_23p] <http://twitter.com/trinityconsults> [YouTube icon_23p] <http://www.youtube.com/trinityconsultants> [https://corporate.trinityconsultants.com/Departments/Marketing/Community%20…] -- _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________

1 0

Re: [mpich-discuss] How To Limit Memory for each MPI Process
by Lu, Huiwei 12 Sep '14

12 Sep '14

Hi, Steven, You mentioned the mapping between a container and a MPI process is one-to-one, right? If you limit the memory usage of a container, then you limit the memory usage of a *process*. Why is that not enough? The “-Xmx” can only be applied to process 0? There should be a way to apply “-Xmx” to all processes. >> All the processes are forked by Hydra Process Manager Proxies, which is not controlled by the container. I am also guessing what you really want is to limit is the memory usage on a *node* that have multiple processes. If so, you can control two things: 1. the memory usage of a container which control the memory usage of each process; 2. the number of processes on a node. Then you should be able to control the the max memory usage of a node. Best, — Huiwei On Sep 12, 2014, at 4:24 AM, Antonio J. Peña <apenya(a)mcs.anl.gov> wrote: > > Hi Steven, > > Sorry, we don't have that functionality built-in in MPICH. You'll have to find an external alternative like those you mentioned. > > Best, > Antonio > > > On 09/12/2014 01:09 AM, Bowen Yu wrote: >> Hi, >> >> It's easy to control the memory usage of Java containers. Specify java parameters like -Xmx when launch the containers and JVM will assure the maximum memory usage. However, limitation on containers is not enough. When the Application Master invokes mpiexec, Hydra will spawn proxy to all the nodes needed by this application via ssh. All the processes are forked by Hydra Process Manager Proxies, which is not controlled by the container. >> >> I don't know if MPI has mechanism to limit memory for each MPI processes. What I really mean is like I specify MAX_MEMORY_USAGE_PER_PROCESS in environment variable, and I am assured that none of the MPI processes will exceed that amount. >> >> >> ------------------ Original ------------------ >> From: "Lu, Huiwei";<huiweilu(a)mcs.anl.gov>; >> Date: Thu, Sep 11, 2014 08:49 PM >> To: "discuss(a)mpich.org"<discuss(a)mpich.org>; >> Subject: Re: [mpich-discuss] How To Limit Memory for each MPI Process >> >> Hi Steven, >> >> It’s good to know that you have enabled MPICH to work with Hadoop YARN. >> >> So you have a Java container inside a MPI process and want to limit the memory usage of a container. A MPI process is as the same as a system process. And you can limit its resource usage use either ulimit or cgroup. However, if the Java container is not aware of the limit and continue to allocate the memory, it will still exceed the memory usage. I don’t know if I understand the problem correctly. But maybe it’s better to limit the memory usage of a Java container. Is there a way to limit the memory usage of a Java container? >> >> Thanks. >> — >> Huiwei Lu >> Postdoc Appointee >> Mathematics and Computer Science Division, Argonne National Laboratory >> http://www.mcs.anl.gov/~huiweilu/ >> >> On Sep 11, 2014, at 6:55 AM, Bowen Yu <jxb002(a)qq.com> wrote: >> >> > Hi, >> > >> > I'm developing a application that enables MPICH executables running at Hadoop YARN cluster, and most functionality has been finished: https://github.com/alibaba/mpich2-yarn. This MPICH-YARN uses MPICH-3.1.2 to run MPI executable. >> > >> > YARN allocate resources in container, and in one container there are specific amount of memory and CPU virtual cores. MPICH-YARN assumes one MPI process is one-to-one correspondence with one container, so the MPI process' memory should be limited. But I have no idea how. How to do so that when I run mpiexec, each process is running with a limited resource, such as memory, and CPU utilization; and if one of the process' memory exceeds, the MPI whole program fails? >> > >> > I know two ways to implement resource limitation in Linux, one is to use system call in programs or ulimit command in shell; the other is to use cgroup kernel module. >> > >> > Thanks! >> > Steven >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> _______________________________________________ >> discuss mailing list >> discuss(a)mpich.org >> >> To manage subscription options or unsubscribe: >> >> https://lists.mpich.org/mailman/listinfo/discuss > > > -- > Antonio J. Peña > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847 > > apenya(a)mcs.anl.gov > www.mcs.anl.gov/~apenya > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How To Limit Memory for each MPI Process
by "Antonio J. Peña" 12 Sep '14

12 Sep '14

Hi Steven, Sorry, we don't have that functionality built-in in MPICH. You'll have to find an external alternative like those you mentioned. Best, Antonio On 09/12/2014 01:09 AM, Bowen Yu wrote: > Hi, > > It's easy to control the memory usage of Java containers. Specify java > parameters like -Xmx when launch the containers and JVM will assure > the maximum memory usage. However, limitation on containers is not > enough. When the Application Master invokes mpiexec, Hydra will spawn > proxy to all the nodes needed by this application via ssh. All the > processes are forked by Hydra Process Manager Proxies, which is not > controlled by the container. > > I don't know if MPI has mechanism to limit memory for each MPI > processes. What I really mean is like I specify > MAX_MEMORY_USAGE_PER_PROCESS in environment variable, and I am assured > that none of the MPI processes will exceed that amount. > > > ------------------ Original ------------------ > *From: * "Lu, Huiwei";<huiweilu(a)mcs.anl.gov>; > *Date: * Thu, Sep 11, 2014 08:49 PM > *To: * "discuss(a)mpich.org"<discuss(a)mpich.org>; > *Subject: * Re: [mpich-discuss] How To Limit Memory for each MPI Process > > Hi Steven, > > It’s good to know that you have enabled MPICH to work with Hadoop YARN. > > So you have a Java container inside a MPI process and want to limit > the memory usage of a container. A MPI process is as the same as a > system process. And you can limit its resource usage use either ulimit > or cgroup. However, if the Java container is not aware of the limit > and continue to allocate the memory, it will still exceed the memory > usage. I don’t know if I understand the problem correctly. But maybe > it’s better to limit the memory usage of a Java container. Is there a > way to limit the memory usage of a Java container? > > Thanks. > — > Huiwei Lu > Postdoc Appointee > Mathematics and Computer Science Division, Argonne National Laboratory > http://www.mcs.anl.gov/~huiweilu/ > > On Sep 11, 2014, at 6:55 AM, Bowen Yu <jxb002(a)qq.com> wrote: > > > Hi, > > > > I'm developing a application that enables MPICH executables running > at Hadoop YARN cluster, and most functionality has been finished: > https://github.com/alibaba/mpich2-yarn. This MPICH-YARN uses > MPICH-3.1.2 to run MPI executable. > > > > YARN allocate resources in container, and in one container there are > specific amount of memory and CPU virtual cores. MPICH-YARN assumes > one MPI process is one-to-one correspondence with one container, so > the MPI process' memory should be limited. But I have no idea how. How > to do so that when I run mpiexec, each process is running with a > limited resource, such as memory, and CPU utilization; and if one of > the process' memory exceeds, the MPI whole program fails? > > > > I know two ways to implement resource limitation in Linux, one is to > use system call in programs or ulimit command in shell; the other is > to use cgroup kernel module. > > > > Thanks! > > Steven > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya

1 0

Re: [mpich-discuss] How To Limit Memory for each MPI Process
by Bowen Yu 12 Sep '14

12 Sep '14

Hi, It's easy to control the memory usage of Java containers. Specify java parameters like -Xmx when launch the containers and JVM will assure the maximum memory usage. However, limitation on containers is not enough. When the Application Master invokes mpiexec, Hydra will spawn proxy to all the nodes needed by this application via ssh. All the processes are forked by Hydra Process Manager Proxies, which is not controlled by the container. I don't know if MPI has mechanism to limit memory for each MPI processes. What I really mean is like I specify MAX_MEMORY_USAGE_PER_PROCESS in environment variable, and I am assured that none of the MPI processes will exceed that amount. ------------------ Original ------------------ From: "Lu, Huiwei";<huiweilu(a)mcs.anl.gov>; Date: Thu, Sep 11, 2014 08:49 PM To: "discuss(a)mpich.org"<discuss(a)mpich.org>; Subject: Re: [mpich-discuss] How To Limit Memory for each MPI Process Hi Steven, It’s good to know that you have enabled MPICH to work with Hadoop YARN. So you have a Java container inside a MPI process and want to limit the memory usage of a container. A MPI process is as the same as a system process. And you can limit its resource usage use either ulimit or cgroup. However, if the Java container is not aware of the limit and continue to allocate the memory, it will still exceed the memory usage. I don’t know if I understand the problem correctly. But maybe it’s better to limit the memory usage of a Java container. Is there a way to limit the memory usage of a Java container? Thanks. — Huiwei Lu Postdoc Appointee Mathematics and Computer Science Division, Argonne National Laboratory http://www.mcs.anl.gov/~huiweilu/ On Sep 11, 2014, at 6:55 AM, Bowen Yu <jxb002(a)qq.com> wrote: > Hi, > > I'm developing a application that enables MPICH executables running at Hadoop YARN cluster, and most functionality has been finished: https://github.com/alibaba/mpich2-yarn. This MPICH-YARN uses MPICH-3.1.2 to run MPI executable. > > YARN allocate resources in container, and in one container there are specific amount of memory and CPU virtual cores. MPICH-YARN assumes one MPI process is one-to-one correspondence with one container, so the MPI process' memory should be limited. But I have no idea how. How to do so that when I run mpiexec, each process is running with a limited resource, such as memory, and CPU utilization; and if one of the process' memory exceeds, the MPI whole program fails? > > I know two ways to implement resource limitation in Linux, one is to use system call in programs or ulimit command in shell; the other is to use cgroup kernel module. > > Thanks! > Steven > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How To Limit Memory for each MPI Process
by Lu, Huiwei 11 Sep '14

11 Sep '14

Hi Steven, It’s good to know that you have enabled MPICH to work with Hadoop YARN. So you have a Java container inside a MPI process and want to limit the memory usage of a container. A MPI process is as the same as a system process. And you can limit its resource usage use either ulimit or cgroup. However, if the Java container is not aware of the limit and continue to allocate the memory, it will still exceed the memory usage. I don’t know if I understand the problem correctly. But maybe it’s better to limit the memory usage of a Java container. Is there a way to limit the memory usage of a Java container? Thanks. — Huiwei Lu Postdoc Appointee Mathematics and Computer Science Division, Argonne National Laboratory http://www.mcs.anl.gov/~huiweilu/ On Sep 11, 2014, at 6:55 AM, Bowen Yu <jxb002(a)qq.com> wrote: > Hi, > > I'm developing a application that enables MPICH executables running at Hadoop YARN cluster, and most functionality has been finished: https://github.com/alibaba/mpich2-yarn. This MPICH-YARN uses MPICH-3.1.2 to run MPI executable. > > YARN allocate resources in container, and in one container there are specific amount of memory and CPU virtual cores. MPICH-YARN assumes one MPI process is one-to-one correspondence with one container, so the MPI process' memory should be limited. But I have no idea how. How to do so that when I run mpiexec, each process is running with a limited resource, such as memory, and CPU utilization; and if one of the process' memory exceeds, the MPI whole program fails? > > I know two ways to implement resource limitation in Linux, one is to use system call in programs or ulimit command in shell; the other is to use cgroup kernel module. > > Thanks! > Steven > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

How To Limit Memory for each MPI Process
by Bowen Yu 11 Sep '14

11 Sep '14

Hi, I'm developing a application that enables MPICH executables running at Hadoop YARN cluster, and most functionality has been finished: https://github.com/alibaba/mpich2-yarn. This MPICH-YARN uses MPICH-3.1.2 to run MPI executable. YARN allocate resources in container, and in one container there are specific amount of memory and CPU virtual cores. MPICH-YARN assumes one MPI process is one-to-one correspondence with one container, so the MPI process' memory should be limited. But I have no idea how. How to do so that when I run mpiexec, each process is running with a limited resource, such as memory, and CPU utilization; and if one of the process' memory exceeds, the MPI whole program fails? I know two ways to implement resource limitation in Linux, one is to use system call in programs or ulimit command in shell; the other is to use cgroup kernel module. Thanks! Steven

1 0

Re: [mpich-discuss] Error building mpich-3.1.2 on Solaris 10 withgcc-4.9.0
by Siegmar Gross 10 Sep '14

10 Sep '14

Hi Ken, > I was able to reproduce the issue on my machine. Libtool seems to not be > able to handle the rpath value specified in LDFLAGS in combination with > its own rpath setting. Modifying your configure time LDFLAGS like this > should allow you to complete the build: > > LDFLAGS=-m64 -L/usr/lib/amd64 -Wl,-rpath -Wl,/usr/lib/amd64 Yes, I was able to build mpich-master-v3.2a1-5-gb293734d on Solaris Sparc and x86_64 with gcc-4.9.0, when I use LDFLAGS="-m64 -L/usr/lib/sparcv9 -Wl,-rpath -Wl,/usr/lib/sparcv9" and LDFLAGS="-m64 -L/usr/lib/amd64 -Wl,-rpath -Wl,/usr/lib/amd64". Thank you very much for your help. Kind regards Siegmar > I'm going to keep investigating, though, why this only appears in this > specific configuration. > > Ken > > On 09/07/2014 03:58 PM, Siegmar Gross wrote: > > Hi, > > > >>>> Thanks, Siegmar. Are you seeing some errors with ÃÂ¢?omake t > >>>> testingÃÂ¢?ÃÂ or are all tests failing? On some platforms, s > >>>> some tests take longer and hence show up as ÃÂ¢?otimeoutsÃÂ¢?ÃÂ, but > >>>> they are not really hanging. But the number of such tests > >>>> should be small. > >>> > >>> I see some bus errors on Solaris 10 Sparc. I attached the log > >>> files for both Solaris 10 Sparc and x86_64, so that you can > >>> see yourself the results. > >>> > >>> Kind regards > >>> > >>> Siegmar > >>> > >> > >> Can you send the output of: > >> > >> gcc -v > >> g++ -v > >> gfortran -v > >> > >> Just want to double check the build configuration so I can match it on > >> my test setup. > > > > > > Solaris 10 x86_64: > > ================== > > > > sunpc1 root 2 gcc -v > > Using built-in specs. > > COLLECT_GCC=gcc > > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_x86_64/gcc-4.9.0/bin/../libexec/gcc/i386 -pc-solaris2.10/4.9.0/lto-wrapper > > Target: i386-pc-solaris2.10 > > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gmp=/usr/local --with-mpfr=/usr/local > > --with-mpc=/usr/local --with-isl=/usr/local --with-cloog=/usr/local --without-gnu-ld --with-ld=/usr/ccs/bin/ld ABI=32 > > --enable-languages=c,c++,objc,obj-c++,fortran --disable-nls --enable-threads=posix --disable-libgcj --with-system-zlib > > --enable-shared --enable-bootstrap > > Thread model: posix > > gcc version 4.9.0 (GCC) > > > > sunpc1 root 3 g++ -v > > Using built-in specs. > > COLLECT_GCC=g++ > > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_x86_64/gcc-4.9.0/bin/../libexec/gcc/i386 -pc-solaris2.10/4.9.0/lto-wrapper > > Target: i386-pc-solaris2.10 > > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gmp=/usr/local --with-mpfr=/usr/local > > --with-mpc=/usr/local --with-isl=/usr/local --with-cloog=/usr/local --without-gnu-ld --with-ld=/usr/ccs/bin/ld ABI=32 > > --enable-languages=c,c++,objc,obj-c++,fortran --disable-nls --enable-threads=posix --disable-libgcj --with-system-zlib > > --enable-shared --enable-bootstrap > > Thread model: posix > > gcc version 4.9.0 (GCC) > > > > sunpc1 root 4 gfortran -v > > Using built-in specs. > > COLLECT_GCC=gfortran > > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_x86_64/gcc-4.9.0/bin/../libexec/gcc/i386 -pc-solaris2.10/4.9.0/lto-wrapper > > Target: i386-pc-solaris2.10 > > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gmp=/usr/local --with-mpfr=/usr/local > > --with-mpc=/usr/local --with-isl=/usr/local --with-cloog=/usr/local --without-gnu-ld --with-ld=/usr/ccs/bin/ld ABI=32 > > --enable-languages=c,c++,objc,obj-c++,fortran --disable-nls --enable-threads=posix --disable-libgcj --with-system-zlib > > --enable-shared --enable-bootstrap > > Thread model: posix > > gcc version 4.9.0 (GCC) > > sunpc1 root 5 > > > > > > Solaris 10 Sparc: > > ================= > > > > tyr root 2 gcc -v > > Using built-in specs. > > COLLECT_GCC=gcc > > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_sparc/gcc-4.9.0/bin/../libexec/gcc/sparc -sun-solaris2.10/4.9.0/lto-wrapper > > Target: sparc-sun-solaris2.10 > > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gnu-as --with-as=/usr/local/bin/as > > --without-gnu-ld --with-ld=/usr/ccs/bin/ld --enable-nls --with-system-zlib --enable-languages=c,c++,objc,fortran,java > > --enable-java-gc=boehm --enable-multilib --disable-libgcj --enable-bootstrap > > Thread model: posix > > gcc version 4.9.0 (GCC) > > > > tyr root 3 g++ -v > > Using built-in specs. > > COLLECT_GCC=g++ > > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_sparc/gcc-4.9.0/bin/../libexec/gcc/sparc -sun-solaris2.10/4.9.0/lto-wrapper > > Target: sparc-sun-solaris2.10 > > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gnu-as --with-as=/usr/local/bin/as > > --without-gnu-ld --with-ld=/usr/ccs/bin/ld --enable-nls --with-system-zlib --enable-languages=c,c++,objc,fortran,java > > --enable-java-gc=boehm --enable-multilib --disable-libgcj --enable-bootstrap > > Thread model: posix > > gcc version 4.9.0 (GCC) > > > > tyr root 4 gfortran -v > > Using built-in specs. > > COLLECT_GCC=gfortran > > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_sparc/gcc-4.9.0/bin/../libexec/gcc/sparc -sun-solaris2.10/4.9.0/lto-wrapper > > Target: sparc-sun-solaris2.10 > > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gnu-as --with-as=/usr/local/bin/as > > --without-gnu-ld --with-ld=/usr/ccs/bin/ld --enable-nls --with-system-zlib --enable-languages=c,c++,objc,fortran,java > > --enable-java-gc=boehm --enable-multilib --disable-libgcj --enable-bootstrap > > Thread model: posix > > gcc version 4.9.0 (GCC) > > tyr root 5 > > > > > > Kind regards > > > > Siegmar > >

1 0

Re: [mpich-discuss] Error building mpich-3.1.2 on Solaris 10 withgcc-4.9.0
by Kenneth Raffenetti 10 Sep '14

10 Sep '14

Siegmar, I was able to reproduce the issue on my machine. Libtool seems to not be able to handle the rpath value specified in LDFLAGS in combination with its own rpath setting. Modifying your configure time LDFLAGS like this should allow you to complete the build: LDFLAGS=-m64 -L/usr/lib/amd64 -Wl,-rpath -Wl,/usr/lib/amd64 I'm going to keep investigating, though, why this only appears in this specific configuration. Ken On 09/07/2014 03:58 PM, Siegmar Gross wrote: > Hi, > >>>> Thanks, Siegmar. Are you seeing some errors with Ã¢?omake t >>>> testingÃ¢?Â or are all tests failing? On some platforms, s >>>> some tests take longer and hence show up as Ã¢?otimeoutsÃ¢?Â, but >>>> they are not really hanging. But the number of such tests >>>> should be small. >>> >>> I see some bus errors on Solaris 10 Sparc. I attached the log >>> files for both Solaris 10 Sparc and x86_64, so that you can >>> see yourself the results. >>> >>> Kind regards >>> >>> Siegmar >>> >> >> Can you send the output of: >> >> gcc -v >> g++ -v >> gfortran -v >> >> Just want to double check the build configuration so I can match it on >> my test setup. > > > Solaris 10 x86_64: > ================== > > sunpc1 root 2 gcc -v > Using built-in specs. > COLLECT_GCC=gcc > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_x86_64/gcc-4.9.0/bin/../libexec/gcc/i386-pc-solaris2.10/4.9.0/lto-wrapper > Target: i386-pc-solaris2.10 > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gmp=/usr/local --with-mpfr=/usr/local > --with-mpc=/usr/local --with-isl=/usr/local --with-cloog=/usr/local --without-gnu-ld --with-ld=/usr/ccs/bin/ld ABI=32 > --enable-languages=c,c++,objc,obj-c++,fortran --disable-nls --enable-threads=posix --disable-libgcj --with-system-zlib > --enable-shared --enable-bootstrap > Thread model: posix > gcc version 4.9.0 (GCC) > > sunpc1 root 3 g++ -v > Using built-in specs. > COLLECT_GCC=g++ > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_x86_64/gcc-4.9.0/bin/../libexec/gcc/i386-pc-solaris2.10/4.9.0/lto-wrapper > Target: i386-pc-solaris2.10 > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gmp=/usr/local --with-mpfr=/usr/local > --with-mpc=/usr/local --with-isl=/usr/local --with-cloog=/usr/local --without-gnu-ld --with-ld=/usr/ccs/bin/ld ABI=32 > --enable-languages=c,c++,objc,obj-c++,fortran --disable-nls --enable-threads=posix --disable-libgcj --with-system-zlib > --enable-shared --enable-bootstrap > Thread model: posix > gcc version 4.9.0 (GCC) > > sunpc1 root 4 gfortran -v > Using built-in specs. > COLLECT_GCC=gfortran > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_x86_64/gcc-4.9.0/bin/../libexec/gcc/i386-pc-solaris2.10/4.9.0/lto-wrapper > Target: i386-pc-solaris2.10 > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gmp=/usr/local --with-mpfr=/usr/local > --with-mpc=/usr/local --with-isl=/usr/local --with-cloog=/usr/local --without-gnu-ld --with-ld=/usr/ccs/bin/ld ABI=32 > --enable-languages=c,c++,objc,obj-c++,fortran --disable-nls --enable-threads=posix --disable-libgcj --with-system-zlib > --enable-shared --enable-bootstrap > Thread model: posix > gcc version 4.9.0 (GCC) > sunpc1 root 5 > > > Solaris 10 Sparc: > ================= > > tyr root 2 gcc -v > Using built-in specs. > COLLECT_GCC=gcc > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_sparc/gcc-4.9.0/bin/../libexec/gcc/sparc-sun-solaris2.10/4.9.0/lto-wrapper > Target: sparc-sun-solaris2.10 > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gnu-as --with-as=/usr/local/bin/as > --without-gnu-ld --with-ld=/usr/ccs/bin/ld --enable-nls --with-system-zlib --enable-languages=c,c++,objc,fortran,java > --enable-java-gc=boehm --enable-multilib --disable-libgcj --enable-bootstrap > Thread model: posix > gcc version 4.9.0 (GCC) > > tyr root 3 g++ -v > Using built-in specs. > COLLECT_GCC=g++ > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_sparc/gcc-4.9.0/bin/../libexec/gcc/sparc-sun-solaris2.10/4.9.0/lto-wrapper > Target: sparc-sun-solaris2.10 > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gnu-as --with-as=/usr/local/bin/as > --without-gnu-ld --with-ld=/usr/ccs/bin/ld --enable-nls --with-system-zlib --enable-languages=c,c++,objc,fortran,java > --enable-java-gc=boehm --enable-multilib --disable-libgcj --enable-bootstrap > Thread model: posix > gcc version 4.9.0 (GCC) > > tyr root 4 gfortran -v > Using built-in specs. > COLLECT_GCC=gfortran > COLLECT_LTO_WRAPPER=/export2/prog/SunOS_sparc/gcc-4.9.0/bin/../libexec/gcc/sparc-sun-solaris2.10/4.9.0/lto-wrapper > Target: sparc-sun-solaris2.10 > Configured with: ../gcc-4.9.0/configure --prefix=/usr/local/gcc-4.9.0 --with-gnu-as --with-as=/usr/local/bin/as > --without-gnu-ld --with-ld=/usr/ccs/bin/ld --enable-nls --with-system-zlib --enable-languages=c,c++,objc,fortran,java > --enable-java-gc=boehm --enable-multilib --disable-libgcj --enable-bootstrap > Thread model: posix > gcc version 4.9.0 (GCC) > tyr root 5 > > > Kind regards > > Siegmar >

1 0