- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] How to spawn a non-MPI process with MPI_Comm_spawn
by Mccall, Kurt E. (MSFC-EV41) 07 Jan '20

07 Jan '20

Ken, Issue created. Can you suggest any other way to launch non-MPI processes? Thanks, Kurt -----Original Message----- From: Raffenetti, Kenneth J. <raffenet(a)mcs.anl.gov> Sent: Tuesday, January 7, 2020 9:32 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] How to spawn a non-MPI process with MPI_Comm_spawn We do not support such a key at this time, but it seems reasonable to add. Could you create an issue on our GitHub? https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_pmodels_mpi… Thanks, Ken On 1/6/20 9:50 AM, Mccall, Kurt E. (MSFC-EV41) via discuss wrote: > I'm attempting to use MPICH to manage a number of non-MPI processes. > The latter call system() and fork(), which as I understand it may > cause undefined behavior if they were spawned as MPI processes. How > do I use > MPI_Comm_spawn() to launch these processes? > > For example, OpenMpi has a key for the "info" field named > "ompi_non_mpi", which causes a non-MPI process to be spawned. Is > there something similar in MPICH? > > Thanks, > > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_m > ailman_listinfo_discuss&d=DwIF-g&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiM > k5Csz9Zk&r=6cP1IfXu3IZOHSDh_vBqciYiIh4uuVgs1MSi5K7l5fQ&m=H8FYbetRLxBRS > aC9l0ktwX7ok3zyPRdhUlWXZZq9nsc&s=cUS7Q28AAksa_KLLMwTztXCU3BiO0lIEKb-bM > Wpcvfk&e= >

1 0

Re: [mpich-discuss] How to spawn a non-MPI process with MPI_Comm_spawn
by Raffenetti, Kenneth J. 07 Jan '20

07 Jan '20

We do not support such a key at this time, but it seems reasonable to add. Could you create an issue on our GitHub? https://github.com/pmodels/mpich Thanks, Ken On 1/6/20 9:50 AM, Mccall, Kurt E. (MSFC-EV41) via discuss wrote: > I’m attempting to use MPICH to manage a number of non-MPI processes. > The latter call system() and fork(), which as I understand it may cause > undefined behavior if they were spawned as MPI processes. How do I use > MPI_Comm_spawn() to launch these processes? > > For example, OpenMpi has a key for the “info” field named > “ompi_non_mpi”, which causes a non-MPI process to be spawned. Is there > something similar in MPICH? > > Thanks, > > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] How to add user-defined lib in MPICH
by Raffenetti, Kenneth J. 07 Jan '20

07 Jan '20

What is the purpose of the library? What function does it provide for MPICH? Ken On 1/7/20 3:53 AM, wanghao via discuss wrote: > The lib HCOLL is added in the MPICH-3.2.1, so if I have a similarly > user-defined lib, how can I add it in MPICH? Is there an official > operating manual to help me? > > I sincerely hope that the mpich community gets better and better. > > Thank you. > > H.Wang > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

How to add user-defined lib in MPICH
by wanghao 07 Jan '20

07 Jan '20

The lib HCOLL is added in the MPICH-3.2.1, so if I have a similarly user-defined lib,  how can I add it in MPICH? Is there an official operating manual to help me? I sincerely hope that  the mpich community gets better and better. Thank you. H.Wang

1 0

How to spawn a non-MPI process with MPI_Comm_spawn
by Mccall, Kurt E. (MSFC-EV41) 06 Jan '20

06 Jan '20

I'm attempting to use MPICH to manage a number of non-MPI processes. The latter call system() and fork(), which as I understand it may cause undefined behavior if they were spawned as MPI processes. How do I use MPI_Comm_spawn() to launch these processes? For example, OpenMpi has a key for the "info" field named "ompi_non_mpi", which causes a non-MPI process to be spawned. Is there something similar in MPICH? Thanks, Kurt

1 0

Re: [mpich-discuss] Best practices for installation procedures of ib-drivers/ucx/pmix/mpich stack
by Raffenetti, Kenneth J. 12 Dec '19

12 Dec '19

Hi, Both PMIx and UCX are runtime dependencies. They must be available on all nodes where your application runs. Location is mostly a style decision. Many systems use /usr/local for packages installed manually (i.e. not by the system package manager). GPFS is also a fine option, which allows you to install only once. Ken On 12/12/19 8:54 AM, Sajid Ali via discuss wrote: > Hi MPICH-developers, > > I'm trying to help the sysadmins at my univ. cluster install mpich using > spack with ucx and pmix (previously we just let mpich interact with > slurm's pmi2 and as mentioned on the mailing list the old versions of > mpich either failed to read the nodelist properly or were (incorrectly) > configured with default TCP/IP transport protocols). > > After the sysadmins installed pmix-2.2.3 and pmix-3.1.3 from rpm's at > /usr/lib64/pmix-version (and configured slurm to use them), I tried > building mpich in my home folder (on GPFS) but it was giving a build > error as it was picking up some wrong libraries from /usr/lib64. When I > installed pmix using spack in my home directory (on GPFS), I was able to > build mpich-devel against it. > > What is the best location to install pmix and ucx? Does it have to be > available on every node and if yes would it be on the local file system > on each node or GPFS file-system ? Or is it okay to have it installed on > the slurm launcher node alone ? > > Hope this is the right place to ask this question and thanks in advance > for the help! > > > -- > Sajid Ali | PhD Candidate > Applied Physics > Northwestern University > s-sajid-ali.github.io <http://s-sajid-ali.github.io> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Best practices for installation procedures of ib-drivers/ucx/pmix/mpich stack
by Sajid Ali 12 Dec '19

12 Dec '19

Hi MPICH-developers, I'm trying to help the sysadmins at my univ. cluster install mpich using spack with ucx and pmix (previously we just let mpich interact with slurm's pmi2 and as mentioned on the mailing list the old versions of mpich either failed to read the nodelist properly or were (incorrectly) configured with default TCP/IP transport protocols). After the sysadmins installed pmix-2.2.3 and pmix-3.1.3 from rpm's at /usr/lib64/pmix-version (and configured slurm to use them), I tried building mpich in my home folder (on GPFS) but it was giving a build error as it was picking up some wrong libraries from /usr/lib64. When I installed pmix using spack in my home directory (on GPFS), I was able to build mpich-devel against it. What is the best location to install pmix and ucx? Does it have to be available on every node and if yes would it be on the local file system on each node or GPFS file-system ? Or is it okay to have it installed on the slurm launcher node alone ? Hope this is the right place to ask this question and thanks in advance for the help! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] too many ssh connections warning
by Reuti 08 Dec '19

08 Dec '19

Hi Kurt: > Am 07.12.2019 um 16:07 schrieb Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>: > > Thanks Reuti. I assume that by the "task manager interface without ssh", you mean > > $ mpiexec -launcher rsh ... No. If the default compilation doesn't include SLURM support in your case, one has to compile MPICH with: $ ./configure --with-slurm=[PATH] … It should look like: $ ps -e f … 16594 ? Sl 0:00 slurmstepd: [166806.batch] 16599 ? S 0:00 \_ /bin/bash /var/spool/slurm/d/job166806/slurm_script 16755 ? S 0:00 \_ mpiexec ./mpihello 16757 ? Ssl 0:00 \_ /bin/srun -N 2 -n 2 --input none /home/reuti/local/mpich-3.3.2/bin/hydra_pmi_pro 16758 ? S 0:00 \_ /bin/srun -N 2 -n 2 --input none /home/reuti/local/mpich-3.3.2/bin/hydra_pmi 16766 ? Sl 0:00 slurmstepd: [166806.0] 16772 ? S 0:00 \_ /home/reuti/local/mpich-3.3.2/bin/hydra_pmi_proxy --control-port node045:3 16773 ? Rs 0:09 \_ ./mpihello 16774 ? Rs 0:09 \_ ./mpihello … (and on the slave nodes only the second daemon is present) No SSH, no RSH. Hence it's a tight integration into the queuing system. https://slurm.schedmd.com/mpi_guide.html In addition, to change to `srun` as startup one might need (not used in the example above): https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Note_that…. It should work with a cluster which has no `ssh` to the nodes implemented at all (I allow `ssh` to the nodes only for admin staff). === Are the nodes on a private network, i.e. the nodes can't be reached from the Internet? Then one might also discuss pro and cons of allowing `rsh` or not. If even inside a private cluster all communication between the nodes has to be encrypted, I fear neither MPICH nor any other MPI implementation provides this. === AFAIR "fork" was used in times before Hydra as an alternative to start the MPI tasks local on a single machine only. I didn't check for "fork" or "smpd" for some time since Hydra appeared. -- Reuti > I can't use rsh on our cluster due to security concerns. Another launcher option is "fork", but when I tried it, the whole job froze. Does "fork" refer to a specific binary like ssh, or does it refer to the Linux system call? > > Thanks, > Kurt > >>> Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. > >> Then it should be possible to use the task manager interface without `ssh`: > > -----Original Message----- > From: Reuti via discuss <discuss(a)mpich.org> > Sent: Tuesday, December 3, 2019 9:02 AM > To: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> > Cc: Reuti <reuti(a)staff.uni-marburg.de> > Subject: Re: [mpich-discuss] [EXTERNAL] Re: too many ssh connections warning > > Hi: > >> Am 03.12.2019 um 15:45 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: >> >> Reuti, >> >> Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. > > Then it should be possible to use the task manager interface without `ssh`: > > https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.adaptivecomputing.… > > >> I'll check with our sysadmins to see if there are firewall issues. > > This could also be later an issue if MPICH will connect to other machines directly to talk to the already started daemons. > > >> What is PAM? > > https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_… > > Several limits can be set here, depending on your distribution: > > ls /lib64/security/ > > will show the available ones which are installed by default and are then used/configured in /etc/pam.d > > -- Reuti > > >> Hui Zhou, >> >> What do you expect would be making multiple SSH connections to the node? The creation of inter-communicators? Individual MPI_Iprobe/MPI_Isend/MPI_IRecv commands? If you have a guess, that would help me know how to fix the problem. >> >> Kurt >> >> >> -----Original Message----- >> From: Reuti <reuti(a)staff.uni-marburg.de> >> Sent: Monday, December 2, 2019 3:20 PM >> To: discuss(a)mpich.org >> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> >> Subject: [EXTERNAL] Re: [mpich-discuss] too many ssh connections warning >> >> >>> Am 02.12.2019 um 22:14 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: >>> >>> My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning >>> >>> [mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds >>> >>> What is the cause of this? I have several guesses: >>> >>> 1) MPICH has an internal limit on the number of connections >>> 2) I’m bumping up against a Linux limit on the number of connections >>> 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) >> >> 4) Firewall or PAM settings on the target prevent to many logins in a certain timeframe. >> >> Are you using a queuing system and have the chance to skip SSH and startup MPICH by the queuing system? >> >> -- Reuti >> >> >>> The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? >>> >>> I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma… >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma… > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…

1 0

Re: [mpich-discuss] too many ssh connections warning
by Mccall, Kurt E. (MSFC-EV41) 07 Dec '19

07 Dec '19

Thanks Reuti. I assume that by the "task manager interface without ssh", you mean $ mpiexec -launcher rsh ... I can't use rsh on our cluster due to security concerns. Another launcher option is "fork", but when I tried it, the whole job froze. Does "fork" refer to a specific binary like ssh, or does it refer to the Linux system call? Thanks, Kurt >> Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. > Then it should be possible to use the task manager interface without `ssh`: -----Original Message----- From: Reuti via discuss <discuss(a)mpich.org> Sent: Tuesday, December 3, 2019 9:02 AM To: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Cc: Reuti <reuti(a)staff.uni-marburg.de> Subject: Re: [mpich-discuss] [EXTERNAL] Re: too many ssh connections warning Hi: > Am 03.12.2019 um 15:45 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: > > Reuti, > > Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. Then it should be possible to use the task manager interface without `ssh`: https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.adaptivecomputing.… > I'll check with our sysadmins to see if there are firewall issues. This could also be later an issue if MPICH will connect to other machines directly to talk to the already started daemons. > What is PAM? https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_… Several limits can be set here, depending on your distribution: ls /lib64/security/ will show the available ones which are installed by default and are then used/configured in /etc/pam.d -- Reuti > Hui Zhou, > > What do you expect would be making multiple SSH connections to the node? The creation of inter-communicators? Individual MPI_Iprobe/MPI_Isend/MPI_IRecv commands? If you have a guess, that would help me know how to fix the problem. > > Kurt > > > -----Original Message----- > From: Reuti <reuti(a)staff.uni-marburg.de> > Sent: Monday, December 2, 2019 3:20 PM > To: discuss(a)mpich.org > Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> > Subject: [EXTERNAL] Re: [mpich-discuss] too many ssh connections warning > > >> Am 02.12.2019 um 22:14 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: >> >> My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning >> >> [mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds >> >> What is the cause of this? I have several guesses: >> >> 1) MPICH has an internal limit on the number of connections >> 2) I’m bumping up against a Linux limit on the number of connections >> 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) > > 4) Firewall or PAM settings on the target prevent to many logins in a certain timeframe. > > Are you using a queuing system and have the chance to skip SSH and startup MPICH by the queuing system? > > -- Reuti > > >> The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? >> >> I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma… > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma… _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…

1 0

Re: [mpich-discuss] What is the best way to connect to a spawned process?
by Mccall, Kurt E. (MSFC-EV41) 05 Dec '19

05 Dec '19

Hui, Thanks for the reply. I am spawning the new process using MPI_COMM_SELF instead of MPI_COMM_WORLD. Maybe that is my problem! Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Wednesday, December 4, 2019 1:43 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] What is the best way to connect to a spawned process? Hi Kurt, Let’s say you have a communicator of 10 processes (MPI_COMM_WORLD or any derivatives), and if you use that communicator to spawn process P, the result inter-communicator will be between your original comm (10 processes) and P’s MPI_COMM_WORLD (1 process), right? All original 10 processes should be already connected to P. Did I miss anything? — Hui Zhou On Dec 4, 2019, at 1:31 PM, Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, My job creates 10 processes, and the rank 0 one spawns a distinct process P to function as a master. I’d like to share P’s inter-communicator with the other 9 processes, but it seems that I can’t pass it directly to them through function arguments to those processes (very weird errors happen involving the inter-communicator). What is an efficient way to connect all 10 processes to P? I’ve looked into MPI_Open_port / MPI_Comm_accept / MPI_Comm_connect, but was wondering if there is more streamlined way. (I can’t make rank 0 become the master directly, due to the peculiarities of my PBS/Torque node allocations). Thanks, Kurt _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…>

1 0