May 2017 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Unable to subscribe to mailing list
by Jeff Hammond 31 May '17

31 May '17

Have you checked your spam folder for the subscription confirmation email? I am subscribed to this list using a gmail handle with a similar construction, so I doubt the system objects to your handle or host, although I've probably been subscribed for a decade at this point. Jeff On Wed, May 31, 2017 at 9:01 AM, Alex Rast <adr1r17(a)soton.ac.uk> wrote: > Dear sirs, > > I've only subscribed under this e-mail address in absolute desperation, > because the one I actually need to use: alex.rast.technical(a)gmail.com, > doesn't seem to work with the system. I've tried subscribing multiple times > with this approach only to receive absolute silence and no indication > either that something went wrong. > > What needs to be done to get the above-referenced e-mail subscribed to the > list? > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Unable to subscribe to mailing list
by Alex Rast 31 May '17

31 May '17

Dear sirs, I've only subscribed under this e-mail address in absolute desperation, because the one I actually need to use: alex.rast.technical(a)gmail.com, doesn't seem to work with the system. I've tried subscribing multiple times with this approach only to receive absolute silence and no indication either that something went wrong. What needs to be done to get the above-referenced e-mail subscribed to the list? _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fwd: MPICH fault tolerance and resiliency
by sanjeev s 30 May '17

30 May '17

Hi, I have added the detailed use case.. I did not follow what were you trying to do here. Can you be more specific? Suppose I have an application which accepts some request from client and do the MPI job on request arrival(lazy load of mpi lib). Now I started this application through mpiexec on 4 machines. *mpiexec will start my application on 4 machines not MPI jobs*. On request arrival I will distribute MPI jobs to this cluster after doing some processing in my application on master node.(1 of out of 4 I will designate as master node). Now I want to add one more machine( having my application) to this cluster, without stopping my already running applications. My master process should be able to see this 5th machine and distribute the task. Is there any way to achieve this? > Also, When I took the size(number of instance for that comm), I am not > getting the count for client instance. To distribute the task, I need to > have additional logic to handle this case in my application. > > Can you be more specific about which comm you were referring to? > > I am referring to MPI_COMM_WORLD. Is there any resize kind of api which will give me how many jobs are there on this comm after doing one accept connection? Hope this helps in understanding the problem. Regards Sanjeev On Mon, May 29, 2017 at 8:48 AM, sanjeev s <snjv.workmail(a)gmail.com> wrote: > Hi, > > Thanks for the reply. I have given the detailed use case below. > > Regards > Sanjeev > > > > On Sat, May 27, 2017 at 2:30 AM, Guo, Yanfei <yguo(a)anl.gov> wrote: > >> Hi Sanjeev, >> >> Please see the inline comments below. >> >> Yanfei Guo >> Postdoctoral Researcher >> MCS Division, ANL >> >> >> On 5/26/17, 10:40 AM, "sanjeev s" <snjv.workmail(a)gmail.com> wrote: >> >> > Hi, >> >> >> > In dynamic process, I read about two models: Client server and parent >> child. >> >> >> > Client Server : We need to have dedicated threads each for client and >> server. Now considering all instances same , we will end up doing lot of >> thread creation apart from our application worker threads. >> >> I assume you are referring to the connect/accept model by client/server. >> In that case, you may need one thread on the server side to handle the >> incoming connection because the `MPI_Comm_accept` call is blocking. A >> dedicated thread is not necessary in the client. It is your >> application’s job to decide who is the server. >> >> > Yes I am referring to same connect/accept model. I don't want to put this > logic in our application. I want some external process manager ( like > Hydra) which can handle this form me .Please see below for detail use case. > > > >> > Moreover when 1 instance (app) goes down, we want >> that instance to come up without doing much manual work. We don't want >> to club this logic in our application. >> >> I did not follow what were you trying to do here. Can you be more >> specific? >> >> Suppose I have an application which accepts some request from client and > do the MPI job on request arrival(lazy load of mpi lib). Now I started this > application through mpiexec on 4 machines. *mpiexec will start my > application on 4 machines not MPI jobs*. On request arrival I will > distribute MPI jobs to this cluster after doing some processing in my > application on master node.(1 of out of 4 I will designate as master > node). Now I want to add one more machine( having my application) to this > cluster, without stopping my already running applications. My master > process should be able to see this 5th machine and distribute the task. Is > there any way to achieve this? > > > > Also, When I took the size(number of instance for that comm), I am not >> getting the count for client instance. To distribute the task, I need to >> have additional logic to handle this case in my application. >> >> Can you be more specific about which comm you were referring to? >> >> I am referring to MPI_COMM_WORLD. Is there any resize kind of api which > will give me how many jobs are there on this comm after doing one accept > connection? > > >> > 2) Parent child: Suppose we have started 4 instance on 4 different >> machines. Now we need to add another server. I don't think parent child/ >> client server is good option in this case. >> >> >> > We don't want to build process management capabilities in our >> application. We are looking for process management in MPI itself (e.g in >> Hydra )so that we can leverage on that. >> >> >> > Please correct me if I am missing something in my understanding of >> Dynamic model. >> >> >> Regards >> Sanjeev Sinha >> >> >> >> >> >> On Fri, May 26, 2017 at 8:46 PM, Halim Amer >> <aamer(a)anl.gov> wrote: >> >> Sanjeev, >> >> > More precisely my requirement is suppose I started 4 instances of my >> > application. Now I want to add one more instance dynamically to this set >> >> From my understanding, dynamic processes would work fine for this case. >> Could you elaborate on why the dynamic process model is not sufficient for >> your needs? >> >> Halim >> www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/~aamer> >> >> On 5/26/17 9:11 AM, sanjeev s wrote: >> >> >> Hi mpich, >> >> I have a requirement where in we need to add start stop application >> instances on the fly before starting a job.Is there any mpich service >> available. I looked through dynamic process model, but its not sufficing >> our need. >> >> More precisely my requirement is suppose I started 4 instances of my >> application. Now I want to add one more instance dynamically to this set >> >> Is there any tool which MPICH supports for fault tolerance behavior? >> >> Thanks >> Sanjeev >> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > >

1 0

Re: [mpich-discuss] Fwd: MPICH fault tolerance and resiliency
by sanjeev s 28 May '17

28 May '17

Hi, Thanks for the reply. I have given the detailed use case below. Regards Sanjeev On Sat, May 27, 2017 at 2:30 AM, Guo, Yanfei <yguo(a)anl.gov> wrote: > Hi Sanjeev, > > Please see the inline comments below. > > Yanfei Guo > Postdoctoral Researcher > MCS Division, ANL > > > On 5/26/17, 10:40 AM, "sanjeev s" <snjv.workmail(a)gmail.com> wrote: > > > Hi, > > > > In dynamic process, I read about two models: Client server and parent > child. > > > > Client Server : We need to have dedicated threads each for client and > server. Now considering all instances same , we will end up doing lot of > thread creation apart from our application worker threads. > > I assume you are referring to the connect/accept model by client/server. > In that case, you may need one thread on the server side to handle the > incoming connection because the `MPI_Comm_accept` call is blocking. A > dedicated thread is not necessary in the client. It is your application’s > job to decide who is the server. > > Yes I am referring to same connect/accept model. I don't want to put this logic in our application. I want some external process manager ( like Hydra) which can handle this form me .Please see below for detail use case. > > Moreover when 1 instance (app) goes down, we want > that instance to come up without doing much manual work. We don't want to > club this logic in our application. > > I did not follow what were you trying to do here. Can you be more > specific? > > Suppose I have an application which accepts some request from client and do the MPI job on request arrival(lazy load of mpi lib). Now I started this application through mpiexec on 4 machines. *mpiexec will start my application on 4 machines not MPI jobs*. On request arrival I will distribute MPI jobs to this cluster after doing some processing in my application on master node.(1 of out of 4 I will designate as master node). Now I want to add one more machine( having my application) to this cluster, without stopping my already running applications. My master process should be able to see this 5th machine and distribute the task. Is there any way to achieve this? > Also, When I took the size(number of instance for that comm), I am not > getting the count for client instance. To distribute the task, I need to > have additional logic to handle this case in my application. > > Can you be more specific about which comm you were referring to? > > I am referring to MPI_COMM_WORLD. Is there any resize kind of api which will give me how many jobs are there on this comm after doing one accept connection? > > 2) Parent child: Suppose we have started 4 instance on 4 different > machines. Now we need to add another server. I don't think parent child/ > client server is good option in this case. > > > > We don't want to build process management capabilities in our > application. We are looking for process management in MPI itself (e.g in > Hydra )so that we can leverage on that. > > > > Please correct me if I am missing something in my understanding of > Dynamic model. > > > Regards > Sanjeev Sinha > > > > > > On Fri, May 26, 2017 at 8:46 PM, Halim Amer > <aamer(a)anl.gov> wrote: > > Sanjeev, > > > More precisely my requirement is suppose I started 4 instances of my > > application. Now I want to add one more instance dynamically to this set > > From my understanding, dynamic processes would work fine for this case. > Could you elaborate on why the dynamic process model is not sufficient for > your needs? > > Halim > www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/~aamer> > > On 5/26/17 9:11 AM, sanjeev s wrote: > > > Hi mpich, > > I have a requirement where in we need to add start stop application > instances on the fly before starting a job.Is there any mpich service > available. I looked through dynamic process model, but its not sufficing > our need. > > More precisely my requirement is suppose I started 4 instances of my > application. Now I want to add one more instance dynamically to this set > > Is there any tool which MPICH supports for fault tolerance behavior? > > Thanks > Sanjeev > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Fwd: MPICH fault tolerance and resiliency
by Guo, Yanfei 26 May '17

26 May '17

Hi Sanjeev, Please see the inline comments below. Yanfei Guo Postdoctoral Researcher MCS Division, ANL On 5/26/17, 10:40 AM, "sanjeev s" <snjv.workmail(a)gmail.com> wrote: > Hi, > In dynamic process, I read about two models: Client server and parent child. > Client Server : We need to have dedicated threads each for client and server. Now considering all instances same , we will end up doing lot of thread creation apart from our application worker threads. I assume you are referring to the connect/accept model by client/server. In that case, you may need one thread on the server side to handle the incoming connection because the `MPI_Comm_accept` call is blocking. A dedicated thread is not necessary in the client. It is your application’s job to decide who is the server. > Moreover when 1 instance (app) goes down, we want that instance to come up without doing much manual work. We don't want to club this logic in our application. I did not follow what were you trying to do here. Can you be more specific? > Also, When I took the size(number of instance for that comm), I am not getting the count for client instance. To distribute the task, I need to have additional logic to handle this case in my application. Can you be more specific about which comm you were referring to? > 2) Parent child: Suppose we have started 4 instance on 4 different machines. Now we need to add another server. I don't think parent child/ client server is good option in this case. > We don't want to build process management capabilities in our application. We are looking for process management in MPI itself (e.g in Hydra )so that we can leverage on that. > Please correct me if I am missing something in my understanding of Dynamic model. Regards Sanjeev Sinha On Fri, May 26, 2017 at 8:46 PM, Halim Amer <aamer(a)anl.gov> wrote: Sanjeev, > More precisely my requirement is suppose I started 4 instances of my > application. Now I want to add one more instance dynamically to this set From my understanding, dynamic processes would work fine for this case. Could you elaborate on why the dynamic process model is not sufficient for your needs? Halim www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/~aamer> On 5/26/17 9:11 AM, sanjeev s wrote: Hi mpich, I have a requirement where in we need to add start stop application instances on the fly before starting a job.Is there any mpich service available. I looked through dynamic process model, but its not sufficing our need. More precisely my requirement is suppose I started 4 instances of my application. Now I want to add one more instance dynamically to this set Is there any tool which MPICH supports for fault tolerance behavior? Thanks Sanjeev _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fwd: MPICH fault tolerance and resiliency
by Halim Amer 26 May '17

26 May '17

I still cannot follow what you are trying to do. The terminology is confusing (what are "app instance", machine, etc.). Using MPI/Hydra terminology would help. Maybe MPI process instead of instance and cluster node instead of machine? Also, I would like to know how MPI terminology maps to the models you described. Do you mean using comm_spawn for parent-child and comm_connect/comm_accept for the client-server model? Having some source code would be much more productive too. Halim www.mcs.anl.gov/~aamer On 5/26/17 10:40 AM, sanjeev s wrote: > Hi, > > In dynamic process, I read about two models: Client server and parent child. > > Client Server : We need to have dedicated threads each for client and > server. Now considering all instances same , we will end up doing lot of > thread creation apart from our application worker threads. Moreover when 1 > instance (app) goes down, we want that instance to come up without doing > much manual work. We don't want to club this logic in our application. > Also, When I took the size(number of instance for that comm), I am not > getting the count for client instance. To distribute the task, I need to > have additional logic to handle this case in my application. > > 2) Parent child: Suppose we have started 4 instance on 4 different > machines. Now we need to add another server. I don't think parent child/ > client server is good option in this case. > > We don't want to build process management capabilities in our application. > We are looking for process management in MPI itself (e.g in Hydra )so that > we can leverage on that. > > Please correct me if I am missing something in my understanding of Dynamic > model. > > Regards > Sanjeev Sinha > > > > On Fri, May 26, 2017 at 8:46 PM, Halim Amer <aamer(a)anl.gov> wrote: > >> Sanjeev, >> >>> More precisely my requirement is suppose I started 4 instances of my >>> application. Now I want to add one more instance dynamically to this set >> >> From my understanding, dynamic processes would work fine for this case. >> Could you elaborate on why the dynamic process model is not sufficient for >> your needs? >> >> Halim >> www.mcs.anl.gov/~aamer >> >> >> On 5/26/17 9:11 AM, sanjeev s wrote: >> >>> Hi mpich, >>> >>> I have a requirement where in we need to add start stop application >>> instances on the fly before starting a job.Is there any mpich service >>> available. I looked through dynamic process model, but its not sufficing >>> our need. >>> >>> More precisely my requirement is suppose I started 4 instances of my >>> application. Now I want to add one more instance dynamically to this set >>> >>> Is there any tool which MPICH supports for fault tolerance behavior? >>> >>> Thanks >>> Sanjeev >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] ULFM revoke doesn't work properly?
by Guo, Yanfei 26 May '17

26 May '17

Hi Nils, The hanging in revoke+shrink is a known issue (see https://github.com/pmodels/mpich/issues/2198) ULFM is still experimental, but we will keep working on it. Best, Yanfei Guo Postdoctoral Researcher MCS Division, ANL On 5/18/17, 5:03 AM, "Nils-Arne Dreier" <n.dreier(a)uni-muenster.de> wrote: Dear MPICH community. I'm currently playing around with the ULFM features for fault-tolerance. I know that these features are experimental, but want to discuss the following example. I used the following minimal example, which deadlock for all nonzero ranks at MPI_Ssend. For my understanding MPI_Ssend should return MPI_ERR_REVOKED, shouldn't it? If i substitute MPI_Ssend with MPI_Send. All ranks reach the MPIX_Comm_shrink command but then deadlock. I observed very rarly that the shrink succeed, but can't determine the reason. #include <iostream> #include <mpi.h> void checkMPIresult(int result){ if(result!=MPI_SUCCESS){ int len; char msg[MPI_MAX_ERROR_STRING]; MPI_Error_string(result, msg, &len); std::cout << msg << std::endl; } } int main(int argc, char** argv){ checkMPIresult(MPI_Init(&argc,&argv)); MPI_Comm comm, new_comm; checkMPIresult(MPI_Comm_dup(MPI_COMM_WORLD,&comm)); MPI_Comm_set_errhandler(comm, MPI_ERRORS_RETURN); int rank = -1; checkMPIresult(MPI_Comm_rank(comm,&rank)); int error = 1; if(rank==0){ std::cout << "revoking..." << std::endl; checkMPIresult(MPIX_Comm_revoke(comm)); error = 0; }else{ checkMPIresult(MPI_Ssend(&error,1,MPI_INT,0,0,comm)); } std::cout << rank << "\twaiting for agree..." << std::endl; checkMPIresult(MPIX_Comm_agree(comm,&error)); std::cout << rank << "\tagreed on " << error << std::endl; checkMPIresult(MPIX_Comm_shrink(comm,&new_comm)); std::cout << rank << "\tcomm shrinked" << std::endl; checkMPIresult(MPI_Comm_free(&new_comm)); checkMPIresult(MPI_Comm_free(&comm)); checkMPIresult(MPI_Finalize()); return 0; } I compile with mpicxx -std=c++11 -pthread mpich-shrink.cc -o mpich-shrink and run with mpirun -n 4 --disable-auto-cleanup ./mpich-shrink I used the recent master branch: The output of mpirun --version is: HYDRA build details: Version: 3.3a2 Release Date: unreleased development copy CC: gcc CXX: g++ F77: gfortran F90: gfortran Configure options: '--disable-option-checking' '--prefix=/home/nils/tmp/mpich-git-install' '--enable-error-checking=all' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/nils/tmp/mpich/src/mpl/include -I/home/nils/tmp/mpich/src/mpl/include -I/home/nils/tmp/mpich/src/openpa/src -I/home/nils/tmp/mpich/src/openpa/src -D_REENTRANT -I/home/nils/tmp/mpich/src/mpi/romio/include' 'MPLLIBNAME=mpl' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Checkpointing libraries available: blcr Demux engines available: poll select Thank you Nils _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fwd: MPICH fault tolerance and resiliency
by sanjeev s 26 May '17

26 May '17

Hi, In dynamic process, I read about two models: Client server and parent child. Client Server : We need to have dedicated threads each for client and server. Now considering all instances same , we will end up doing lot of thread creation apart from our application worker threads. Moreover when 1 instance (app) goes down, we want that instance to come up without doing much manual work. We don't want to club this logic in our application. Also, When I took the size(number of instance for that comm), I am not getting the count for client instance. To distribute the task, I need to have additional logic to handle this case in my application. 2) Parent child: Suppose we have started 4 instance on 4 different machines. Now we need to add another server. I don't think parent child/ client server is good option in this case. We don't want to build process management capabilities in our application. We are looking for process management in MPI itself (e.g in Hydra )so that we can leverage on that. Please correct me if I am missing something in my understanding of Dynamic model. Regards Sanjeev Sinha On Fri, May 26, 2017 at 8:46 PM, Halim Amer <aamer(a)anl.gov> wrote: > Sanjeev, > > > More precisely my requirement is suppose I started 4 instances of my > > application. Now I want to add one more instance dynamically to this set > > From my understanding, dynamic processes would work fine for this case. > Could you elaborate on why the dynamic process model is not sufficient for > your needs? > > Halim > www.mcs.anl.gov/~aamer > > > On 5/26/17 9:11 AM, sanjeev s wrote: > >> Hi mpich, >> >> I have a requirement where in we need to add start stop application >> instances on the fly before starting a job.Is there any mpich service >> available. I looked through dynamic process model, but its not sufficing >> our need. >> >> More precisely my requirement is suppose I started 4 instances of my >> application. Now I want to add one more instance dynamically to this set >> >> Is there any tool which MPICH supports for fault tolerance behavior? >> >> Thanks >> Sanjeev >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Fwd: MPICH fault tolerance and resiliency
by Halim Amer 26 May '17

26 May '17

Sanjeev, > More precisely my requirement is suppose I started 4 instances of my > application. Now I want to add one more instance dynamically to this set From my understanding, dynamic processes would work fine for this case. Could you elaborate on why the dynamic process model is not sufficient for your needs? Halim www.mcs.anl.gov/~aamer On 5/26/17 9:11 AM, sanjeev s wrote: > Hi mpich, > > I have a requirement where in we need to add start stop application > instances on the fly before starting a job.Is there any mpich service > available. I looked through dynamic process model, but its not sufficing > our need. > > More precisely my requirement is suppose I started 4 instances of my > application. Now I want to add one more instance dynamically to this set > > Is there any tool which MPICH supports for fault tolerance behavior? > > Thanks > Sanjeev > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Fwd: MPICH fault tolerance and resiliency
by sanjeev s 26 May '17

26 May '17

Hi mpich, I have a requirement where in we need to add start stop application instances on the fly before starting a job.Is there any mpich service available. I looked through dynamic process model, but its not sufficing our need. More precisely my requirement is suppose I started 4 instances of my application. Now I want to add one more instance dynamically to this set Is there any tool which MPICH supports for fault tolerance behavior? Thanks Sanjeev _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0