November 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Error when using mpiexec whith mpich-3.04
by Rajeev Thakur 21 Nov '13

21 Nov '13

statut has to be allocated of size MPI_STATUS_SIZE not nb_pr. On Nov 21, 2013, at 10:29 AM, vincent russier <russier(a)icmpe.cnrs.fr> wrote: > I include the trial code as an attached file. In the meantime, I tried > to run with increasing number of processes; I realized that it works > without error messages if np >= 8 and fails otherwise. > > Thanks, > VR > > Le jeudi 21 novembre 2013 à 10:03 -0600, Wesley Bland a écrit : >> Can you include the minimum working example? We need to see the code to know what's going wrong. >> >> Thanks, >> Wesley >> >>> On Nov 21, 2013, at 10:00 AM, vincent russier <russier(a)icmpe.cnrs.fr> wrote: >>> >>> Hi, >>> >>> I have installed mpich-3.04 on a computer powered by linux. >>> Everything seems to be OK. The example in f.90 runs correctecly >>> without error mesages. When I try my own trial code (simply a send >>> and a recv call) I get the following error mesage : >>> >>> Assertion failed in file src/mpid/ch3/src/mpid_vc.c at line 140: >>> MPIU_Object_get_ref((vcrt)) >= 0 >>> >>> The process which aborts is the one on which I use the recv >>> instruction. >>> >>> I have use mpiexec -n <n> ./trial.out >>> >>> I have tried with ifort and gfortran compilers with the same result. >>> >>> I have made the same install on another computer also powered by >>> Linux with exactly the same result. >>> >>> The 2 Linux are different: Fedora 19 and RedHat Enterprise 6.2 >>> >>> Thank you for your help. >>> VR >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > <test_mpi.f90>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH Page Hacked
by Marcelo Soares Souza 21 Nov '13

21 Nov '13

Hi, Apparently the main mpich.org was cracked.

1 0

Re: [mpich-discuss] Error when using mpiexec whith mpich-3.04
by vincent russier 21 Nov '13

21 Nov '13

I include the trial code as an attached file. In the meantime, I tried to run with increasing number of processes; I realized that it works without error messages if np >= 8 and fails otherwise. Thanks, VR Le jeudi 21 novembre 2013 à 10:03 -0600, Wesley Bland a écrit : > Can you include the minimum working example? We need to see the code to know what's going wrong. > > Thanks, > Wesley > > > On Nov 21, 2013, at 10:00 AM, vincent russier <russier(a)icmpe.cnrs.fr> wrote: > > > > Hi, > > > > I have installed mpich-3.04 on a computer powered by linux. > > Everything seems to be OK. The example in f.90 runs correctecly > > without error mesages. When I try my own trial code (simply a send > > and a recv call) I get the following error mesage : > > > > Assertion failed in file src/mpid/ch3/src/mpid_vc.c at line 140: > > MPIU_Object_get_ref((vcrt)) >= 0 > > > > The process which aborts is the one on which I use the recv > > instruction. > > > > I have use mpiexec -n <n> ./trial.out > > > > I have tried with ifort and gfortran compilers with the same result. > > > > I have made the same install on another computer also powered by > > Linux with exactly the same result. > > > > The 2 Linux are different: Fedora 19 and RedHat Enterprise 6.2 > > > > Thank you for your help. > > VR > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Error when using mpiexec whith mpich-3.04
by vincent russier 21 Nov '13

21 Nov '13

Hi, I have installed mpich-3.04 on a computer powered by linux. Everything seems to be OK. The example in f.90 runs correctecly without error mesages. When I try my own trial code (simply a send and a recv call) I get the following error mesage : Assertion failed in file src/mpid/ch3/src/mpid_vc.c at line 140: MPIU_Object_get_ref((vcrt)) >= 0 The process which aborts is the one on which I use the recv instruction. I have use mpiexec -n <n> ./trial.out I have tried with ifort and gfortran compilers with the same result. I have made the same install on another computer also powered by Linux with exactly the same result. The 2 Linux are different: Fedora 19 and RedHat Enterprise 6.2 Thank you for your help. VR

1 0

Re: [mpich-discuss] Error when using mpiexec whith mpich-3.04
by Wesley Bland 21 Nov '13

21 Nov '13

Can you include the minimum working example? We need to see the code to know what's going wrong. Thanks, Wesley > On Nov 21, 2013, at 10:00 AM, vincent russier <russier(a)icmpe.cnrs.fr> wrote: > > Hi, > > I have installed mpich-3.04 on a computer powered by linux. > Everything seems to be OK. The example in f.90 runs correctecly > without error mesages. When I try my own trial code (simply a send > and a recv call) I get the following error mesage : > > Assertion failed in file src/mpid/ch3/src/mpid_vc.c at line 140: > MPIU_Object_get_ref((vcrt)) >= 0 > > The process which aborts is the one on which I use the recv > instruction. > > I have use mpiexec -n <n> ./trial.out > > I have tried with ifort and gfortran compilers with the same result. > > I have made the same install on another computer also powered by > Linux with exactly the same result. > > The 2 Linux are different: Fedora 19 and RedHat Enterprise 6.2 > > Thank you for your help. > VR > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [mvapich-discuss] Non-MPI_THREAD-SINGLE mode with enabled MV2 affinity?
by Ken Raffenetti 21 Nov '13

21 Nov '13

Hi, Sorry for the delayed response on this. We have an open ticket to improve the hydra docs on our wiki, and it should be completed before the 3.1 release. One thing you can do (this should be clearer in the docs) is use -help after a hydra option to see additional information on the usage. For example, 'mpiexec -bind-to -help' will display more process binding info and some examples. Ken ----- Original Message ----- > From: "Jonathan Perkins" <perkinjo(a)cse.ohio-state.edu> > To: "Thiago Quirino - NOAA Federal" <thiago.quirino(a)noaa.gov>, discuss(a)mpich.org > Cc: "mvapich-discuss" <mvapich-discuss(a)cse.ohio-state.edu> > Sent: Friday, November 15, 2013 12:42:04 PM > Subject: Re: [mpich-discuss] [mvapich-discuss] Non-MPI_THREAD-SINGLE mode with enabled MV2 affinity? > > Thanks for the suggestion. This wiki is actually maintained by the > MPICH group at ANL. I'm including their discuss list in this reply. > > I believe that the biggest improvement to the wiki would be to expand > on the syntax accepted by user defined binding. > > On Fri, Nov 15, 2013 at 1:01 PM, Thiago Quirino - NOAA Federal > <thiago.quirino(a)noaa.gov> wrote: > > Thank you so much, Jonathan. > > > > Is it possible to add this information to the Wiki? Though I've > > searched the > > net for hours, unfortunately I couldn't find any reference to this. > > The > > closest I found regarding disabling internal MVAPICH2 and > > mpiexec.hydra > > affinities in favor of external MPI process pinning using taskset > > or numactl > > was this: > > > > https://scicomp.jlab.org/docs/node/66 > > > > Thank you so much again, Jonathan. > > Thiago. > > > > > > > > > > On Fri, Nov 15, 2013 at 10:31 AM, Jonathan Perkins > > <perkinjo(a)cse.ohio-state.edu> wrote: > >> > >> Hello Thiago. I've confirmed that the following syntax will give > >> you > >> what you're asking for. You'll basically need to specify user > >> binding, replace your commas with pluses, and replace your colons > >> with > >> commas. > >> > >> [perkinjo@sandy1 install]$ ./bin/mpiexec -n 4 -bind-to > >> user:0+1+2+3,4+5+6+7,8+9+10+11,12+13+14+15 -env > >> MV2_ENABLE_AFFINITY=0 > >> ./cpumask > >> 1111000000000000 > >> 0000111100000000 > >> 0000000011110000 > >> 0000000000001111 > >> > >> On Wed, Nov 13, 2013 at 7:02 PM, Thiago Quirino - NOAA Federal > >> <thiago.quirino(a)noaa.gov> wrote: > >> > Hi, Jonathan. > >> > > >> > Using mpiexec.hydra's binding capability, is it possible to > >> > assign a CPU > >> > range for each MPI task in a node? Suppose I want to spawn 4 MPI > >> > tasks > >> > per > >> > node, where each node has 2 sockets with 8 cores each (total 16 > >> > cores). > >> > I > >> > want task 1 to run on CPU range 0-3, task 2 on CPU range 4-7, > >> > task 3 on > >> > CPU > >> > range 8-11, and task 4 on CPU range 12-15. I used to accomplish > >> > this > >> > using > >> > the MV2_CPU_BINDING variables as follows: > >> > > >> > export MV2_CPU_MAPPING=0,1,2,3:4,5,6,7:8,9,10,11:12,13,14,15 > >> > > >> > Can I accomplish the same binding configuration with > >> > mpiexec.hydra's > >> > binding > >> > capability? I only see socket binding options in the Wiki. > >> > > >> > Thanks again, Jonathan. > >> > Thiago. > >> > > >> > > >> > > >> > On Tue, Nov 12, 2013 at 1:56 PM, Jonathan Perkins > >> > <jonathan.lamar.perkins(a)gmail.com> wrote: > >> >> > >> >> Hello Thiago. Perhaps you can try an alternative to > >> >> MV2_ENABLE_AFFINITY. If you use the hydra process manager > >> >> (mpiexec.hydra), you can disable the library affinity and use > >> >> the > >> >> launcher affinity instead. In this case the other threading > >> >> levels > >> >> will be available to you. > >> >> > >> >> Please see > >> >> > >> >> https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Proc… > >> >> for more information on how to use this hydra feature. Also > >> >> please do > >> >> not forget to set MV2_ENABLE_AFFINITY to 0. > >> >> > >> >> Please let us know if this helps. > >> >> > >> >> On Fri, Nov 8, 2013 at 6:41 PM, Thiago Quirino - NOAA Federal > >> >> <thiago.quirino(a)noaa.gov> wrote: > >> >> > Hi, folks. Quick question about MVAPICH2 and affinity > >> >> > support. > >> >> > > >> >> > Is it possible to invoke MPI_Init_thread with any mode other > >> >> > than > >> >> > "MPI_THREAD_SINGLE" and still use "MV2_ENABLE_AFFINITY=1"? In > >> >> > my > >> >> > hybrid > >> >> > application I mix MPI with raw Pthreads (not OpenMP). I start > >> >> > 4 MPI > >> >> > tasks in > >> >> > each 16 cores node, where each node has 2 sockets with 8 > >> >> > Sandybridge > >> >> > cores > >> >> > each. Each of the 4 MPI tasks then spawns 4 pthreads for a > >> >> > total of > >> >> > 16 > >> >> > pthreads/node, or 1 pthread/core. Within each MPI task, the > >> >> > MPI calls > >> >> > are > >> >> > serialized among the 4 pthreads, so I can use any > >> >> > MPI_THREAD_* mode, > >> >> > but > >> >> > I > >> >> > don't know which mode will work best. I want to assign each > >> >> > of the 4 > >> >> > MPI > >> >> > tasks in a node a set of 4 cores using MV2_CPU_MAPPING (e.g. > >> >> > export > >> >> > MV2_CPU_MAPPING=0,1,2,3:4,5,6,7:8,9,10,11:12,13,14,15) so > >> >> > that the 4 > >> >> > pthreads spawned by each MPI task can migrate to any > >> >> > processor within > >> >> > its > >> >> > exclusive CPU set of size 4. > >> >> > > >> >> > Is that possible with modes other than MPI_THREAD_SINGLE? If > >> >> > not, do > >> >> > you > >> >> > foresee any issues with using MPI_THREAD_SINGLE while > >> >> > serializing the > >> >> > MPI > >> >> > calls among the 4 pthreads of each MPI task? That is, is > >> >> > there any > >> >> > advantage > >> >> > to using MPI_THREAD_FUNELLED or MPI_THREAD_SERIALIZED versus > >> >> > MPI_THREAD_SINGLE for serialized calls among pthreads? > >> >> > > >> >> > Thank you so much, folks. Any help is much appreciated. > >> >> > > >> >> > Best, > >> >> > Thiago. > >> >> > > >> >> > > >> >> > --------------------------------------------------- > >> >> > Thiago Quirino, Ph.D. > >> >> > NOAA Hurricane Research Division > >> >> > 4350 Rickenbacker Cswy. > >> >> > Miami, FL 33139 > >> >> > P: 305-361-4503 > >> >> > E: Thiago.Quirino(a)noaa.gov > >> >> > > >> >> > _______________________________________________ > >> >> > mvapich-discuss mailing list > >> >> > mvapich-discuss(a)cse.ohio-state.edu > >> >> > http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Jonathan Perkins > >> > > >> > > >> > > >> > _______________________________________________ > >> > mvapich-discuss mailing list > >> > mvapich-discuss(a)cse.ohio-state.edu > >> > http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > >> > > >> > >> > >> > >> -- > >> Jonathan Perkins > >> http://www.cse.ohio-state.edu/~perkinjo > >> > > > > > > -- > Jonathan Perkins > http://www.cse.ohio-state.edu/~perkinjo > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Predefined datatype implementation
by Jed Brown 20 Nov '13

20 Nov '13

Jeff Hammond <jeff.science(a)gmail.com> writes: > On Wed, Nov 20, 2013 at 2:20 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: >> Jeff Hammond <jeff.science(a)gmail.com> writes: >> >>>> 1. System headers and libraries are still often on a remote filesystem. >>>> The workflow is a lot more complicated if you mirror that locally. >>> >>> The workflow does not change at all. E.g. "#include <stdio.h>" works >>> irrespective of where you build. >> >> Yes, slowly because it's coming from the global file system. > > Sorry, but no, /usr, /opt or /soft are not mounted from a global file > system. The beauty of static linkage on this huge supercomputers is > that the system headers and objects don't have to be visible to the > compute nodes. When they are, it is usually (e.g. Cray and IBM Blue > Gene) via a special ramdisk or local mount that makes the access fast. Depends on the system, but on ALCF systems /projects/ is GPFS. That includes /projects/llvm/, among many others. That's what I get when I load +mpiwrapper-bgclang. >> 1. You have to remember an extra number (the login node number). > > Clearly, this is too great a burden for any computational scientist to bear. Even experts make mistakes, and every extra thing slows down and complicates the process. > Novices don't build anything from source, just like novice drives > don't chair their own oil or tires. You're abusing the word novice in > an attempt to make a weak argument if you contend otherwise. Lots of people, from grad students to full professors at top-10 research universities and senior scientists at national laboratories trip over this stuff on a regular basis. In this context, "novice" can mean anyone without significant experience in compilation, linking, and HPC systems. >> Or we can agree that although requiring every package to test a zillion >> things in their configure sucks for workflow, usability, and speed, >> there is no viable technical solution available, so we have few options >> short of the life-consuming task of attempting to build and distribute a >> real solution. > > Your novice users should use the provided installations or request one > if the package they need is not available. If they want more than > release versions of standard packages, they aren't novices and need to > RTFM their way out of their problems. I do not believe that ALCF or any similar facility has the manpower to install all packages that novice (non-expert) users require. Meanwhile, many of those packages don't have quality release processes, and it's not uncommon for upstream to have no experience on a particular architecture. And smaller university clusters have much less support resources. The fact is that a large fraction (likely the majority) of computational science is performed by people that are not experts at the practicalities of running on HPC systems. Unless you intend to overhaul the education system, this will continue to be the case and perhaps escalate. Overhauling education would be a laudable goal, but will take a lot of effort and a lot of time to effect. Meanwhile, we live in the present.

1 0

Re: [mpich-discuss] Predefined datatype implementation
by Jeff Hammond 20 Nov '13

20 Nov '13

On Wed, Nov 20, 2013 at 2:20 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Jeff Hammond <jeff.science(a)gmail.com> writes: > >>> 1. System headers and libraries are still often on a remote filesystem. >>> The workflow is a lot more complicated if you mirror that locally. >> >> The workflow does not change at all. E.g. "#include <stdio.h>" works >> irrespective of where you build. > > Yes, slowly because it's coming from the global file system. Sorry, but no, /usr, /opt or /soft are not mounted from a global file system. The beauty of static linkage on this huge supercomputers is that the system headers and objects don't have to be visible to the compute nodes. When they are, it is usually (e.g. Cray and IBM Blue Gene) via a special ramdisk or local mount that makes the access fast. >>> 2. /tmp isn't shared between login nodes, let alone compute nodes, so >>> you have to copy back to global storage. With multiple login nodes, >>> when you background/nohup a compilation task and log out or are >>> disconnected, you have to remember the login node number to get back to >>> the result. The cognitive load of doing all compilation on /tmp, >>> especially with multiple packages, is not trivial. >> >> This is not complicated. You just ssh back to the login node you were >> building on before or you rsync between them. People that don't know >> how to use ssh and rsync shouldn't be building MPICH from source. > > 1. You have to remember an extra number (the login node number). Clearly, this is too great a burden for any computational scientist to bear. > 2. Normal users don't build MPICH from source on expensive machines, but > they do build PETSc and downstream packages. Since there is no standard > system for caching the results of configure tests, each test will be run > at each relevant node within the dependency graph, which is a lot for > applications with a deep stack consisting of well-factored libraries. > >>> 3. Try teaching this to a novice. The fact is that if HPC is going to >>> grow its user base, it has to be easier to use. If you want to take the >>> elitist stance that expensive machines should only be used by experts, >>> stop giving INCITE awards to people that don't know how to use >>> compilers, debuggers, or the shell. Until then, hiccups along the way >>> land in our lap, and a complicated workflow only serves to intimidate >>> and to create more opportunity for human error, which again comes back >>> in the form of support mail. >> >> Novices don't build MPICH from source. > > The whole question is about configure tests for packages depending on > MPI, not for building the MPI implementation. Novices don't build anything from source, just like novice drives don't chair their own oil or tires. You're abusing the word novice in an attempt to make a weak argument if you contend otherwise. >> I like fish, but I refuse to swallow all the red herrings you keep >> feeding me. > > Does this mean you are volunteering to take over the support email for > incompetent INCITE awardees and that you are happy to alienate the > scientists with modest technical skills that are not yet able to win an > INCITE? TMI for this list. > Or we can agree that although requiring every package to test a zillion > things in their configure sucks for workflow, usability, and speed, > there is no viable technical solution available, so we have few options > short of the life-consuming task of attempting to build and distribute a > real solution. Your novice users should use the provided installations or request one if the package they need is not available. If they want more than release versions of standard packages, they aren't novices and need to RTFM their way out of their problems. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] Predefined datatype implementation
by Jed Brown 20 Nov '13

20 Nov '13

Jeff Hammond <jeff.science(a)gmail.com> writes: >> 1. System headers and libraries are still often on a remote filesystem. >> The workflow is a lot more complicated if you mirror that locally. > > The workflow does not change at all. E.g. "#include <stdio.h>" works > irrespective of where you build. Yes, slowly because it's coming from the global file system. >> 2. /tmp isn't shared between login nodes, let alone compute nodes, so >> you have to copy back to global storage. With multiple login nodes, >> when you background/nohup a compilation task and log out or are >> disconnected, you have to remember the login node number to get back to >> the result. The cognitive load of doing all compilation on /tmp, >> especially with multiple packages, is not trivial. > > This is not complicated. You just ssh back to the login node you were > building on before or you rsync between them. People that don't know > how to use ssh and rsync shouldn't be building MPICH from source. 1. You have to remember an extra number (the login node number). 2. Normal users don't build MPICH from source on expensive machines, but they do build PETSc and downstream packages. Since there is no standard system for caching the results of configure tests, each test will be run at each relevant node within the dependency graph, which is a lot for applications with a deep stack consisting of well-factored libraries. >> 3. Try teaching this to a novice. The fact is that if HPC is going to >> grow its user base, it has to be easier to use. If you want to take the >> elitist stance that expensive machines should only be used by experts, >> stop giving INCITE awards to people that don't know how to use >> compilers, debuggers, or the shell. Until then, hiccups along the way >> land in our lap, and a complicated workflow only serves to intimidate >> and to create more opportunity for human error, which again comes back >> in the form of support mail. > > Novices don't build MPICH from source. The whole question is about configure tests for packages depending on MPI, not for building the MPI implementation. > I like fish, but I refuse to swallow all the red herrings you keep > feeding me. Does this mean you are volunteering to take over the support email for incompetent INCITE awardees and that you are happy to alienate the scientists with modest technical skills that are not yet able to win an INCITE? Or we can agree that although requiring every package to test a zillion things in their configure sucks for workflow, usability, and speed, there is no viable technical solution available, so we have few options short of the life-consuming task of attempting to build and distribute a real solution.

1 0

Re: [mpich-discuss] Predefined datatype implementation
by Jeff Hammond 20 Nov '13

20 Nov '13

>> Filesystem performance is a red herring. Don't compile on the >> gold-plated filesystem. Use /tmp or /scratch, as suggested by >> https://www.alcf.anl.gov/user-guides/compiling-and-linking-faq. Every >> Linux login node has /tmp. > > 1. System headers and libraries are still often on a remote filesystem. > The workflow is a lot more complicated if you mirror that locally. The workflow does not change at all. E.g. "#include <stdio.h>" works irrespective of where you build. > 2. /tmp isn't shared between login nodes, let alone compute nodes, so > you have to copy back to global storage. With multiple login nodes, > when you background/nohup a compilation task and log out or are > disconnected, you have to remember the login node number to get back to > the result. The cognitive load of doing all compilation on /tmp, > especially with multiple packages, is not trivial. This is not complicated. You just ssh back to the login node you were building on before or you rsync between them. People that don't know how to use ssh and rsync shouldn't be building MPICH from source. > 3. Try teaching this to a novice. The fact is that if HPC is going to > grow its user base, it has to be easier to use. If you want to take the > elitist stance that expensive machines should only be used by experts, > stop giving INCITE awards to people that don't know how to use > compilers, debuggers, or the shell. Until then, hiccups along the way > land in our lap, and a complicated workflow only serves to intimidate > and to create more opportunity for human error, which again comes back > in the form of support mail. Novices don't build MPICH from source. I like fish, but I refuse to swallow all the red herrings you keep feeding me. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0