February 2017 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI error
by Seth Munholland 15 Feb '17

15 Feb '17

That's very strange as I never installed OpenMPI. I used apt-get to install MPICH and have avoided OpenMPI to try and not get this issue. I thought it would be an easy fix of uninstalling and reinstalling via apt-get, but I can't seem to actually remove this OpenMPI version. Would a source compile write over the apt-get installed version? Seth Munholland, B.Sc. Department of Biological Sciences Rm. 304 Biology Building University of Windsor 401 Sunset Ave. N9B 3P4 T: (519) 253-3000 Ext: 4755 On Wed, Feb 8, 2017 at 11:50 AM, Halim Amer <halim.amer(a)acm.org> wrote: > Hi, > > It seems you are using Open MPI. You can whether contact the Open MPI team > to look into your problem, or clean your environment (both binaries and > libraries) if you want to use MPICH to make sure you are truly using MPICH > when building and running your program. > > Halim > www.mcs.anl.gov/~aamer > > > On 2/8/17 10:18 AM, Seth Munholland wrote: > >> Hello everyone, >> >> I have been configuring a new ubuntu cluster and wanted it to run MPI >> programs. I got mpich configured and compiled then ran the following as >> a test: >> >> #include <stdio.h> >> #include <mpi.h> >> >> int main (int argc, char** argv) { >> int rank = 0, size = 0, nameLen = 0; >> char procName[MPI_MAX_PROCESSOR_NAME]; >> >> MPI_Init (&argc, &argv); >> MPI_Comm_size (MPI_COMM_WORLD, &size); >> MPI_Comm_rank (MPI_COMM_WORLD, &rank); >> MPI_Get_processor_name (procName, &nameLen); >> >> printf ("Hello from processor %s, rank %d of %d\n", procName, rank, >> size); >> >> MPI_Finalize(); >> return 0; >> } >> >> using the commands: >> mpicc mpi_hello.c -o mpi_hello >> mpiexec -mahcinefile machinefile mpi_hello >> >> my machienfile looks like this: >> beanblade4:24 >> beanblade3:24 >> beanblade2:24 >> beanblade:24 >> >> Which it properly defined in /etc/hosts. It compiled and executed >> without error and returned all the processes I had on each node. I went >> on to install some programs on the NFS drive and when I tried to run it >> I get the following error >> >> bash: orted: command not found >> ------------------------------------------------------------ >> -------------- >> ORTE was unable to reliably start one or more daemons. >> This usually is caused by: >> >> * not finding the required libraries and/or binaries on >> one or more nodes. Please check your PATH and LD_LIBRARY_PATH >> settings, or configure OMPI with --enable-orterun-prefix-by-default >> >> * lack of authority to execute on one or more specified nodes. >> Please verify your allocation and authorities. >> >> * the inability to write startup files into /tmp >> (--tmpdir/orte_tmpdir_base). >> Please check with your sys admin to determine the correct location to >> use. >> >> * compilation of the orted with dynamic libraries when static are >> required >> (e.g., on Cray). Please check your configure cmd line and consider using >> one of the contrib/platform definitions for your system type. >> >> * an inability to create a connection back to mpirun due to a >> lack of common network interfaces and/or no route found between >> them. Please check network connectivity (including firewalls >> and network routing requirements). >> ------------------------------------------------------------ >> -------------- >> >> This error now gets returned when I try to run the mpi_hello program I >> used to test my mpi implementation. if I drop the machinefile tag from >> my mpiexec command I get >> >> mpiexec ./mpi_hello >> ------------------------------------------------------------ >> -------------- >> [[14894,1],17]: A high-performance Open MPI point-to-point messaging >> module >> was unable to find any relevant network interfaces: >> >> Module: OpenFabrics (openib) >> Host: beanblade >> >> Another transport will be used instead, although this may result in >> lower performance. >> ------------------------------------------------------------ >> -------------- >> Hello from processor beanblade, rank 18 of 24 >> Hello from processor beanblade, rank 0 of 24 >> Hello from processor beanblade, rank 2 of 24 >> Hello from processor beanblade, rank 4 of 24 >> Hello from processor beanblade, rank 6 of 24 >> Hello from processor beanblade, rank 7 of 24 >> Hello from processor beanblade, rank 3 of 24 >> Hello from processor beanblade, rank 1 of 24 >> Hello from processor beanblade, rank 5 of 24 >> Hello from processor beanblade, rank 8 of 24 >> Hello from processor beanblade, rank 12 of 24 >> Hello from processor beanblade, rank 13 of 24 >> Hello from processor beanblade, rank 11 of 24 >> Hello from processor beanblade, rank 9 of 24 >> Hello from processor beanblade, rank 10 of 24 >> Hello from processor beanblade, rank 14 of 24 >> Hello from processor beanblade, rank 15 of 24 >> Hello from processor beanblade, rank 16 of 24 >> Hello from processor beanblade, rank 17 of 24 >> Hello from processor beanblade, rank 19 of 24 >> Hello from processor beanblade, rank 20 of 24 >> Hello from processor beanblade, rank 21 of 24 >> Hello from processor beanblade, rank 22 of 24 >> Hello from processor beanblade, rank 23 of 24 >> [beanblade:13849] 23 more processes have sent help message >> help-mpi-btl-base.txt / btl:no-nics >> [beanblade:13849] Set MCA parameter "orte_base_help_aggregate" to 0 to >> see all help / error messages >> >> I've been trying to figure it out on the forums and I think it's >> somethign to do with my bashrc file (I have a few exports for >> environmental variables), but removing them didn't fix the problem. >> What did I break? Where do I look to fix it? >> >> Seth Munholland, B.Sc. >> Department of Biological Sciences >> Rm. 304 Biology Building >> University of Windsor >> 401 Sunset Ave. N9B 3P4 >> T: (519) 253-3000 Ext: 4755 >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How does hyday works?
by Balaji, Pavan 14 Feb '17

14 Feb '17

It's a distributed service. -- Pavan > On Feb 14, 2017, at 2:23 PM, Dorier, Matthieu <mdorier(a)anl.gov> wrote: > > Is the key-value server located on the machine that ran mpiexec, or on one of the machine in the hostfile? > Thanks, > > Matthieu > ________________________________________ > From: Balaji, Pavan [balaji(a)anl.gov] > Sent: Tuesday, February 14, 2017 2:04 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] How does hyday works? > > Well, not exactly. The environment variables tell MPICH how to connect to a key-value server. Then each process puts its business card (e.g., IP address and port) into the key-value server and all processes can query this information for the target process they want to communicate with. > > -- Pavan > >> On Feb 14, 2017, at 11:19 AM, Dorier, Matthieu <mdorier(a)anl.gov> wrote: >> >> Hi, >> >> I'd like to have a better understanding of how Hydra works and how MPI processes "discover" how to connect to one another. I'm guessing something roughly like this happens: >> >> - mpiexec launches and ssh connection to each node, and sets some environment variables that will indicate to the created processes how to connect to one another; >> - MPI processes are created, and when calling MPI_Init they read the right environment variables to build MPI_COMM_WORLD. >> >> Is this correct? If so, where could I find information about the environment variables at play? >> >> Thanks, >> >> Matthieu >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How does hyday works?
by Dorier, Matthieu 14 Feb '17

14 Feb '17

Is the key-value server located on the machine that ran mpiexec, or on one of the machine in the hostfile? Thanks, Matthieu ________________________________________ From: Balaji, Pavan [balaji(a)anl.gov] Sent: Tuesday, February 14, 2017 2:04 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] How does hyday works? Well, not exactly. The environment variables tell MPICH how to connect to a key-value server. Then each process puts its business card (e.g., IP address and port) into the key-value server and all processes can query this information for the target process they want to communicate with. -- Pavan > On Feb 14, 2017, at 11:19 AM, Dorier, Matthieu <mdorier(a)anl.gov> wrote: > > Hi, > > I'd like to have a better understanding of how Hydra works and how MPI processes "discover" how to connect to one another. I'm guessing something roughly like this happens: > > - mpiexec launches and ssh connection to each node, and sets some environment variables that will indicate to the created processes how to connect to one another; > - MPI processes are created, and when calling MPI_Init they read the right environment variables to build MPI_COMM_WORLD. > > Is this correct? If so, where could I find information about the environment variables at play? > > Thanks, > > Matthieu > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How does hyday works?
by Balaji, Pavan 14 Feb '17

14 Feb '17

Well, not exactly. The environment variables tell MPICH how to connect to a key-value server. Then each process puts its business card (e.g., IP address and port) into the key-value server and all processes can query this information for the target process they want to communicate with. -- Pavan > On Feb 14, 2017, at 11:19 AM, Dorier, Matthieu <mdorier(a)anl.gov> wrote: > > Hi, > > I'd like to have a better understanding of how Hydra works and how MPI processes "discover" how to connect to one another. I'm guessing something roughly like this happens: > > - mpiexec launches and ssh connection to each node, and sets some environment variables that will indicate to the created processes how to connect to one another; > - MPI processes are created, and when calling MPI_Init they read the right environment variables to build MPI_COMM_WORLD. > > Is this correct? If so, where could I find information about the environment variables at play? > > Thanks, > > Matthieu > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

How does hyday works?
by Dorier, Matthieu 14 Feb '17

14 Feb '17

Hi, I'd like to have a better understanding of how Hydra works and how MPI processes "discover" how to connect to one another. I'm guessing something roughly like this happens: - mpiexec launches and ssh connection to each node, and sets some environment variables that will indicate to the created processes how to connect to one another; - MPI processes are created, and when calling MPI_Init they read the right environment variables to build MPI_COMM_WORLD. Is this correct? If so, where could I find information about the environment variables at play? Thanks, Matthieu _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI error
by Halim Amer 08 Feb '17

08 Feb '17

Hi, It seems you are using Open MPI. You can whether contact the Open MPI team to look into your problem, or clean your environment (both binaries and libraries) if you want to use MPICH to make sure you are truly using MPICH when building and running your program. Halim www.mcs.anl.gov/~aamer On 2/8/17 10:18 AM, Seth Munholland wrote: > Hello everyone, > > I have been configuring a new ubuntu cluster and wanted it to run MPI > programs. I got mpich configured and compiled then ran the following as > a test: > > #include <stdio.h> > #include <mpi.h> > > int main (int argc, char** argv) { > int rank = 0, size = 0, nameLen = 0; > char procName[MPI_MAX_PROCESSOR_NAME]; > > MPI_Init (&argc, &argv); > MPI_Comm_size (MPI_COMM_WORLD, &size); > MPI_Comm_rank (MPI_COMM_WORLD, &rank); > MPI_Get_processor_name (procName, &nameLen); > > printf ("Hello from processor %s, rank %d of %d\n", procName, rank, > size); > > MPI_Finalize(); > return 0; > } > > using the commands: > mpicc mpi_hello.c -o mpi_hello > mpiexec -mahcinefile machinefile mpi_hello > > my machienfile looks like this: > beanblade4:24 > beanblade3:24 > beanblade2:24 > beanblade:24 > > Which it properly defined in /etc/hosts. It compiled and executed > without error and returned all the processes I had on each node. I went > on to install some programs on the NFS drive and when I tried to run it > I get the following error > > bash: orted: command not found > -------------------------------------------------------------------------- > ORTE was unable to reliably start one or more daemons. > This usually is caused by: > > * not finding the required libraries and/or binaries on > one or more nodes. Please check your PATH and LD_LIBRARY_PATH > settings, or configure OMPI with --enable-orterun-prefix-by-default > > * lack of authority to execute on one or more specified nodes. > Please verify your allocation and authorities. > > * the inability to write startup files into /tmp > (--tmpdir/orte_tmpdir_base). > Please check with your sys admin to determine the correct location to use. > > * compilation of the orted with dynamic libraries when static are required > (e.g., on Cray). Please check your configure cmd line and consider using > one of the contrib/platform definitions for your system type. > > * an inability to create a connection back to mpirun due to a > lack of common network interfaces and/or no route found between > them. Please check network connectivity (including firewalls > and network routing requirements). > -------------------------------------------------------------------------- > > This error now gets returned when I try to run the mpi_hello program I > used to test my mpi implementation. if I drop the machinefile tag from > my mpiexec command I get > > mpiexec ./mpi_hello > -------------------------------------------------------------------------- > [[14894,1],17]: A high-performance Open MPI point-to-point messaging module > was unable to find any relevant network interfaces: > > Module: OpenFabrics (openib) > Host: beanblade > > Another transport will be used instead, although this may result in > lower performance. > -------------------------------------------------------------------------- > Hello from processor beanblade, rank 18 of 24 > Hello from processor beanblade, rank 0 of 24 > Hello from processor beanblade, rank 2 of 24 > Hello from processor beanblade, rank 4 of 24 > Hello from processor beanblade, rank 6 of 24 > Hello from processor beanblade, rank 7 of 24 > Hello from processor beanblade, rank 3 of 24 > Hello from processor beanblade, rank 1 of 24 > Hello from processor beanblade, rank 5 of 24 > Hello from processor beanblade, rank 8 of 24 > Hello from processor beanblade, rank 12 of 24 > Hello from processor beanblade, rank 13 of 24 > Hello from processor beanblade, rank 11 of 24 > Hello from processor beanblade, rank 9 of 24 > Hello from processor beanblade, rank 10 of 24 > Hello from processor beanblade, rank 14 of 24 > Hello from processor beanblade, rank 15 of 24 > Hello from processor beanblade, rank 16 of 24 > Hello from processor beanblade, rank 17 of 24 > Hello from processor beanblade, rank 19 of 24 > Hello from processor beanblade, rank 20 of 24 > Hello from processor beanblade, rank 21 of 24 > Hello from processor beanblade, rank 22 of 24 > Hello from processor beanblade, rank 23 of 24 > [beanblade:13849] 23 more processes have sent help message > help-mpi-btl-base.txt / btl:no-nics > [beanblade:13849] Set MCA parameter "orte_base_help_aggregate" to 0 to > see all help / error messages > > I've been trying to figure it out on the forums and I think it's > somethign to do with my bashrc file (I have a few exports for > environmental variables), but removing them didn't fix the problem. > What did I break? Where do I look to fix it? > > Seth Munholland, B.Sc. > Department of Biological Sciences > Rm. 304 Biology Building > University of Windsor > 401 Sunset Ave. N9B 3P4 > T: (519) 253-3000 Ext: 4755 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI error
by Seth Munholland 08 Feb '17

08 Feb '17

Hello everyone, I have been configuring a new ubuntu cluster and wanted it to run MPI programs. I got mpich configured and compiled then ran the following as a test: #include <stdio.h> #include <mpi.h> int main (int argc, char** argv) { int rank = 0, size = 0, nameLen = 0; char procName[MPI_MAX_PROCESSOR_NAME]; MPI_Init (&argc, &argv); MPI_Comm_size (MPI_COMM_WORLD, &size); MPI_Comm_rank (MPI_COMM_WORLD, &rank); MPI_Get_processor_name (procName, &nameLen); printf ("Hello from processor %s, rank %d of %d\n", procName, rank, size); MPI_Finalize(); return 0; } using the commands: mpicc mpi_hello.c -o mpi_hello mpiexec -mahcinefile machinefile mpi_hello my machienfile looks like this: beanblade4:24 beanblade3:24 beanblade2:24 beanblade:24 Which it properly defined in /etc/hosts. It compiled and executed without error and returned all the processes I had on each node. I went on to install some programs on the NFS drive and when I tried to run it I get the following error bash: orted: command not found -------------------------------------------------------------------------- ORTE was unable to reliably start one or more daemons. This usually is caused by: * not finding the required libraries and/or binaries on one or more nodes. Please check your PATH and LD_LIBRARY_PATH settings, or configure OMPI with --enable-orterun-prefix-by-default * lack of authority to execute on one or more specified nodes. Please verify your allocation and authorities. * the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base). Please check with your sys admin to determine the correct location to use. * compilation of the orted with dynamic libraries when static are required (e.g., on Cray). Please check your configure cmd line and consider using one of the contrib/platform definitions for your system type. * an inability to create a connection back to mpirun due to a lack of common network interfaces and/or no route found between them. Please check network connectivity (including firewalls and network routing requirements). -------------------------------------------------------------------------- This error now gets returned when I try to run the mpi_hello program I used to test my mpi implementation. if I drop the machinefile tag from my mpiexec command I get mpiexec ./mpi_hello -------------------------------------------------------------------------- [[14894,1],17]: A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces: Module: OpenFabrics (openib) Host: beanblade Another transport will be used instead, although this may result in lower performance. -------------------------------------------------------------------------- Hello from processor beanblade, rank 18 of 24 Hello from processor beanblade, rank 0 of 24 Hello from processor beanblade, rank 2 of 24 Hello from processor beanblade, rank 4 of 24 Hello from processor beanblade, rank 6 of 24 Hello from processor beanblade, rank 7 of 24 Hello from processor beanblade, rank 3 of 24 Hello from processor beanblade, rank 1 of 24 Hello from processor beanblade, rank 5 of 24 Hello from processor beanblade, rank 8 of 24 Hello from processor beanblade, rank 12 of 24 Hello from processor beanblade, rank 13 of 24 Hello from processor beanblade, rank 11 of 24 Hello from processor beanblade, rank 9 of 24 Hello from processor beanblade, rank 10 of 24 Hello from processor beanblade, rank 14 of 24 Hello from processor beanblade, rank 15 of 24 Hello from processor beanblade, rank 16 of 24 Hello from processor beanblade, rank 17 of 24 Hello from processor beanblade, rank 19 of 24 Hello from processor beanblade, rank 20 of 24 Hello from processor beanblade, rank 21 of 24 Hello from processor beanblade, rank 22 of 24 Hello from processor beanblade, rank 23 of 24 [beanblade:13849] 23 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics [beanblade:13849] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages I've been trying to figure it out on the forums and I think it's somethign to do with my bashrc file (I have a few exports for environmental variables), but removing them didn't fix the problem. What did I break? Where do I look to fix it? Seth Munholland, B.Sc. Department of Biological Sciences Rm. 304 Biology Building University of Windsor 401 Sunset Ave. N9B 3P4 T: (519) 253-3000 Ext: 4755 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] spurious lock ups on collective merge intercom
by Dmitriy Lyubimov 07 Feb '17

07 Feb '17

On Tue, Feb 7, 2017 at 2:08 PM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> wrote: > > > On Tue, Feb 7, 2017 at 11:15 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > >> Hi Dmitriy, >> >> You should use publish/lookup name for this, instead of relying on manual >> printing. I've attached updated server and client codes that do that. >> Please see attached. >> >> You'll need to start the Hydra nameserver for this using: >> >> % hydra_nameserver >> >> Once the nameserver has started, you can connect all mpiexecs to it using >> something like: >> >> % mpiexec -nameserver localhost ./server 2 >> > > Thanks but we have our own name resolution architecture (which is not > manual printing/pasting). This is simple example that's been asked for to > isolated the problem and provided, not the actual code. > > The name is guaranteed to be delivered to client verbatim down to the bit. > The use is fully consistent with the MPI 3 spec. > Sorry for reiterating on this point. Just checked again, here's the quote from 3.1 MPI standard, MPI_COMM_Connect(): "port_name is the address of the server. It must be the same as the name returned by MPI_OPEN_PORT on the server." So, the standard is very clear about it. For as long as the port name is faithfully communicated from process that opened it to every other process, it does not have to be obtained via any strictly prescribed means of redistribution. And any error in the port name should just raise MPI_ERR_PORT. Which is not what we observe, so I am afraid using or not using MPI_Lookup_name is not the problem / a solution of this issue. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] spurious lock ups on collective merge intercom
by Dmitriy Lyubimov 07 Feb '17

07 Feb '17

also like i said, with 3.3a2 it was much (much!) harder to reproduce... but it still happens from time to time. like i said, stack analysis for 3.3a2 is slightly different. in 3.2 we have one post-connect and (n-1) post-accept stacks all sitting and waiting in intercom_merge(), with 3.3a2 it is slightly different -- we observerd 2 post-connects waiting on merge, which this scheme should not allow at all. When we go thru merging, there should always be one post-connect merge and (n-1) post-accept merge calls. So either one of post-connects is still there from the previous loop (which should not be possible as it would not allow to accept on the current loop), or two clients were accepted somehow at the same time (which should not be the case either, as 3-way intercoms are not possible). But the analysis is very thorough, we get full backtraces of all processes, and their internal state (such as current intracom size before accept) is also known, so we rule out things like process failures etc., i am pretty confident of that. We did that so many times it is fairly difficult to believe we still haven't accounted for naive things like dead processes etc. On Tue, Feb 7, 2017 at 2:10 PM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> wrote: > >> yes. Like i said. I am able to achieve lock up state spuriously on 192 >> core cluster only if i spin up almost all cores (per process) >> > > should read "1 core per process" > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] spurious lock ups on collective merge intercom
by Dmitriy Lyubimov 07 Feb '17

07 Feb '17

> > > yes. Like i said. I am able to achieve lock up state spuriously on 192 > core cluster only if i spin up almost all cores (per process) > should read "1 core per process" _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0