December 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] The max amount of data each process can send in MPI_Alltoallv
by Huiwei Lu 15 Dec '13

15 Dec '13

Thank you for your tips, Rajeev, Jeff and JT, I used Rajeev’s suggestion to send about 4GB data (100M items) per process, and it works fine. @JT, yes, if the data continue to grow and there is no proper way to do it in MPI type, I will have to partition the buffer to smaller trunks. Thanks again for your help. — Huiwei On Dec 13, 2013, at 7:02 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > You can create a contiguous datatype of size 1GB and then send 1G items of that size :-). > > On Dec 13, 2013, at 5:18 PM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: > >> Hi, folks, >> >> What’s the max amount of data each process can send in MPI_Alltoallv? http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Alltoallv.html >> >> As I understand, the input parameter of sendcnts is an array of int, so the amount of data each process can send is limited to MAX_INT (approximately 2G). >> >> What if I want to use MPI_Alltoallv-like MPI routines to send a buffer of data that exceeds 2GB? Does it need to cut the data buffers to smaller sizes? Is there an easy way to do this? The reason I am asking is one of my distributed sort algorithm uses MPI_Alltoallv to distribute data after partitioning; it crashed when I tried to send in each process more than 2GB data. >> >> Thanks. >> -- >> Huiwei Lu >> http://www.mcs.anl.gov/~huiweilu/ >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] The max amount of data each process can send in MPI_Alltoallv
by jt.meng 13 Dec '13

13 Dec '13

Hi, Huiwei, I think the best solution for your question is cuting the data buffers into smaller sizes. However, This may be not easy. According to my experience, the suggestions are 1. Do not use MPI_Alltoallv, and use nonblocking communication insted, such as MPI_send and MPI_irecv. 2. In solution 1, each data buffer can not exceeds 2GB too, If the data is too large, I think we should develop a class encapsulated with the data buffer cuting method. However, using int as the parameters' type in MPI communication primitives is not reasonable. As we know, in x86_64 machine, sizeof(int)=4, however sizeof(long) and sizeof(long long int)=8, we should support the new parameter type of long and long long int. jt.meng From: Huiwei Lu Date: 2013-12-14 07:18 To: discuss Subject: [mpich-discuss] The max amount of data each process can send in MPI_Alltoallv Hi, folks, What’s the max amount of data each process can send in MPI_Alltoallv? http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Alltoallv.html As I understand, the input parameter of sendcnts is an array of int, so the amount of data each process can send is limited to MAX_INT (approximately 2G). What if I want to use MPI_Alltoallv-like MPI routines to send a buffer of data that exceeds 2GB? Does it need to cut the data buffers to smaller sizes? Is there an easy way to do this? The reason I am asking is one of my distributed sort algorithm uses MPI_Alltoallv to distribute data after partitioning; it crashed when I tried to send in each process more than 2GB data. Thanks. -- Huiwei Lu http://www.mcs.anl.gov/~huiweilu/

1 0

Re: [mpich-discuss] The max amount of data each process can send in MPI_Alltoallv
by Rajeev Thakur 13 Dec '13

13 Dec '13

You can create a contiguous datatype of size 1GB and then send 1G items of that size :-). On Dec 13, 2013, at 5:18 PM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: > Hi, folks, > > What’s the max amount of data each process can send in MPI_Alltoallv? http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Alltoallv.html > > As I understand, the input parameter of sendcnts is an array of int, so the amount of data each process can send is limited to MAX_INT (approximately 2G). > > What if I want to use MPI_Alltoallv-like MPI routines to send a buffer of data that exceeds 2GB? Does it need to cut the data buffers to smaller sizes? Is there an easy way to do this? The reason I am asking is one of my distributed sort algorithm uses MPI_Alltoallv to distribute data after partitioning; it crashed when I tried to send in each process more than 2GB data. > > Thanks. > -- > Huiwei Lu > http://www.mcs.anl.gov/~huiweilu/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

The max amount of data each process can send in MPI_Alltoallv
by Huiwei Lu 13 Dec '13

13 Dec '13

Hi, folks, What’s the max amount of data each process can send in MPI_Alltoallv? http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Alltoallv.html As I understand, the input parameter of sendcnts is an array of int, so the amount of data each process can send is limited to MAX_INT (approximately 2G). What if I want to use MPI_Alltoallv-like MPI routines to send a buffer of data that exceeds 2GB? Does it need to cut the data buffers to smaller sizes? Is there an easy way to do this? The reason I am asking is one of my distributed sort algorithm uses MPI_Alltoallv to distribute data after partitioning; it crashed when I tried to send in each process more than 2GB data. Thanks. -- Huiwei Lu http://www.mcs.anl.gov/~huiweilu/

1 0

Re: [mpich-discuss] sctp channel for mpich2
by Pavan Balaji 13 Dec '13

13 Dec '13

Sorry, the sctp channel is no longer provided in mpich. Old versions that provide it are not support. — Pavan On Dec 14, 2013, at 5:27 AM, hamideh jabalameli <hamideh.jabalameli(a)gmail.com> wrote: > > Hi Dear Sir > > I configure MPICH2 with ch3:sctp channel,then the configure has been complete and make & make install done successful. > Now I don't know how use sctp channel for example MPI...? > my command : > > mpiexec -np 2 exampleapp2mpi. but this example doesn't run base to sctp channel > > Can you help me? > > -- > Best Regards > > Hamideh Jabalameli > > M.Sc Candidate, > School of Electrical and Computer Engineering, > College of Engineering, > Electronici Shahid Beheshti Tehran University > h.jabal(a)Mail.sbu.ac.ir > hamideh.jabalameli(a)gmail.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

halo update performance comparison
by Kokron, Daniel S. (GSFC-610.1)[Computer Sciences Corporation] 13 Dec '13

13 Dec '13

All, I have been interested in MPI datatypes since reading about them and playing with the benchmark found at http://unixer.de/research/datatypes/ddtbench/ I am working with a weather model that spends considerable time doing halo (aka ghost cell) updates. The existing code uses irecv/isend. Since most of the time spent doing halo updates is actually spent packing and unpacking the MPI buffers, I was hoping the use of datatypes would improve performance. Unfortunately, the datatype version is actually quite a bit slower than the standard non-blocking pt2pt code. I found this to be that case for both mpich-3.1rc1 and a recent version of a vendor MPI implementation (not MPICH based). I have not run this comparison using OpenMPI. Is data type performance an active area of development? Is there room for improvement? Daniel Kokron NASA Ames (ARC-TN) SciCon group 301-286-3959

1 0

sctp channel for mpich2
by hamideh jabalameli 13 Dec '13

13 Dec '13

Hi Dear Sir I configure MPICH2 with ch3:sctp channel,then the configure has been complete and make & make install done successful. Now I don't know how use sctp channel for example MPI...? my command : mpiexec -np 2 exampleapp2mpi. but this example doesn't run base to sctp channel Can you help me? -- *Best Regards* *Hamideh Jabalameli* M.Sc Candidate, School of Electrical and Computer Engineering, College of Engineering, Electronici Shahid Beheshti Tehran University h.jabal(a)Mail.sbu.ac.ir hamideh.jabalameli(a)gmail.com

1 0

Re: [mpich-discuss] mpifort wrapper compiler?
by William Gropp 13 Dec '13

13 Dec '13

This seems fine. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Dec 13, 2013, at 2:28 AM, Pavan Balaji wrote: > > OK. I’ll run this by BillG as well. I’ve created a ticket: > > https://trac.mpich.org/projects/mpich/ticket/1986 > > It’s a little more complicated than I initially thought (it’s not a simple symbolic link). So we’ll not do this for 3.1, which we are in the process of finalizing. > > — Pavan > > On Dec 12, 2013, at 12:18 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > >> On Dec 12, 2013, at 8:02 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >>> Will the following tradeoff be acceptable to everyone? >>> >>> * We allow users to specify a Fortran compiler using “FC” and “FCFLAGS”. This will result in a wrapper called “mpifort”. >>> >>> * We’ll maintain mpif90 as a symbolic link to mpifort for backward compatibility. >>> >>> * If no separate F77 compiler is specified, we’ll use FC as the F77 compiler as well. In this case, mpif77 will be a symbolic link to mpifort. >>> >>> * But users have the ability to provide a separate F77 compiler, in which case a new mpif77 is created. >>> >>> This way, we can still support compilers that have a different F77 compiler. But for most compilers there’ll be a single mpifort (with mpif90 and mpif77 symbolic links for backward compatibility for some time). >> >> That sounds perfect. >> >> -- >> Jeff Squyres >> jsquyres(a)cisco.com >> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpifort wrapper compiler?
by Pavan Balaji 13 Dec '13

13 Dec '13

OK. I’ll run this by BillG as well. I’ve created a ticket: https://trac.mpich.org/projects/mpich/ticket/1986 It’s a little more complicated than I initially thought (it’s not a simple symbolic link). So we’ll not do this for 3.1, which we are in the process of finalizing. — Pavan On Dec 12, 2013, at 12:18 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > On Dec 12, 2013, at 8:02 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> Will the following tradeoff be acceptable to everyone? >> >> * We allow users to specify a Fortran compiler using “FC” and “FCFLAGS”. This will result in a wrapper called “mpifort”. >> >> * We’ll maintain mpif90 as a symbolic link to mpifort for backward compatibility. >> >> * If no separate F77 compiler is specified, we’ll use FC as the F77 compiler as well. In this case, mpif77 will be a symbolic link to mpifort. >> >> * But users have the ability to provide a separate F77 compiler, in which case a new mpif77 is created. >> >> This way, we can still support compilers that have a different F77 compiler. But for most compilers there’ll be a single mpifort (with mpif90 and mpif77 symbolic links for backward compatibility for some time). > > That sounds perfect. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] MPI memory allocation.
by Anatoly G 12 Dec '13

12 Dec '13

Hi. Finally, I got an additional info. I build short simulation of my real application. *The short description of real scenario.* I have Master + N slaves. Each slave sends to Master 2 types of messages: 1. constant length message with predefined fields (one of it's fields is length of second message). 2. second message - length of this message each time is different and passed in first message. Master should use MPI_Irecv commands, in order to be tolerant to slaves failure (blocked MPI_Recv is blocking Master in the failure case). Master executes MPI_Irecv to each slave with buffer size equal to the constant size of first message type. After receiving first type of message, Master allocates expected buffer for second message and performs receive too. This happens in endless loop for each slaves. I use MPI_Waitany to monitor all receives. In order to separate messages Master & slaves use different tags (as ids) for first & second messages. *Simulation description:* All passed buffers (first & second) have the same size. Slave (SndSyncSlave) sends messages and swaps 2 tags between them (like 2 types of messages, but second one has constant size too). Master routine (Rcv_WaitAny function) executes MPI_Irecv for first message, and after receive executes MPI_Irecv for the second one. In this scenario: 5 processes works fine, but if I execute 20 processes and remove comment from line "usleep(200000)" I see 800 Mbit/s on network at the test beginning, but after 1-2 second network speed become 200-300Kbit/s and not increased back any more. If I add MPI_Recv block in Master (remove comment from "MPI_Recv" and line around) I see that Master starts increase memory as my real application, but again on 5 processes this not happens. This is scenario used in my real application. *Command line:* mpiexec.hydra -genvall -f MpiConfigMachines.txt -launcher=ssh -n 20 mpi_rcv_any_multithread 100000 1000000 out where 100000 - number of sends from each slave 1000000 - scale to separate input from each scale (used for debug only) out - prefix of output file. Each process produce out_"rank".txt file. MpiConfigMachines.txt - configuration file for my computers, 2 computers back to back 1 Gbit/s network. Can you please test this case, and give me yours suggestions. Thank you, Anatoly. On Mon, Dec 9, 2013 at 9:55 PM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > Yes, I understand that. I"ll try to make my stand alone test closer to > real application. Thank you. > > > On Mon, Dec 9, 2013 at 9:31 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> >> It sounds like MPICH is working correctly. Without a test case, it’s >> unfortunately quite hard for us to even know what to look for. It’s also >> possible that there’s a bug in your code which might be causing some bad >> behavior. >> >> — Pavan >> >> On Dec 9, 2013, at 1:27 PM, Anatoly G <anatolyrishon(a)gmail.com> wrote: >> >> > Yes, I"m actually need Fault tolerance, and it was the main reason for >> choosing MPICH2. I use fault tolerance for unpredictable bugs in the >> future. My system should survive partially. But in the regular case I just >> need full performance. I"m suspect that I don't use MPI correctly, but on >> slow rate everything works fine. The fail caused by increasing rate of >> MPI_Isend or increasing data buffer size. I didn't find yet any strong >> dependence, only main stream. >> > >> > Unfortunately I have a complex system which has a number of threads in >> each process. Part of the threads use different communicators. >> > >> > I try to simulate the same MPI behavior in simple stand alone test, but >> stand alone test works fine. It shows a full network performance, when I >> slow down master (on stand alone test), all slaves are stopped too and are >> waiting for master to continue. Can I open any MPICH log to send you the >> results? >> > >> > >> > On Mon, Dec 9, 2013 at 8:10 PM, Pavan Balaji <balaji(a)mcs.anl.gov> >> wrote: >> > >> > Do you actually need Fault Tolerance (one of your previous emails >> seemed to indicate that)? >> > >> > It sounds like there a bug in either your application or in the MPICH >> stack and you are trying to trace that down, and don’t really care about >> fault tolerance. Is that a correct assessment? >> > >> > Do you have a simplified program that reproduces this error, that we >> can try? >> > >> > — Pavan >> > >> > On Dec 9, 2013, at 11:44 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: >> > >> > > No. Hardware is Ok. Master process allocates memory (check with >> MemoryScape doesn't show any sufficient memory allocation in my code). Then >> network traffic become low, and then Master process crashes w/o saving core >> file. I have unlimited size of core files. The same fail (w/o core) I see >> when I call MPI_Abort, but I don't call it. >> > > >> > > >> > > On Mon, Dec 9, 2013 at 7:28 PM, Wesley Bland <wbland(a)mcs.anl.gov> >> wrote: >> > > Are you actually seeing hardware failure or is your code just >> crashing? It's odd that one specific process would fail so often in the >> same way if it were a hardware problem. >> > > >> > > Thanks, >> > > Wesley >> > > >> > > On Dec 9, 2013, at 11:15 AM, Anatoly G <anatolyrishon(a)gmail.com> >> wrote: >> > > >> > >> One more interesting fact. Each time I have a failure, the fails >> only master process, but slaves are still exists together with >> mpiexec.hydra. I thought that slaves should fail too, but slaves are live. >> > >> >> > >> >> > >> On Mon, Dec 9, 2013 at 10:30 AM, Anatoly G <anatolyrishon(a)gmail.com> >> wrote: >> > >> I configure MPICH-3.1rc2 build w/o "so" files. But instead of MPICH2 >> & MPICH-3.0.4 I get so files. What should I change in configure line to >> link MPI with my application statically. >> > >> >> > >> >> > >> >> > >> On Mon, Dec 9, 2013 at 9:47 AM, Pavan Balaji <balaji(a)mcs.anl.gov> >> wrote: >> > >> >> > >> Can you try mpich-3.1rc2? There were several fixes for this in this >> version and it’ll be good to try that out before we go digging too far into >> this. >> > >> >> > >> — Pavan >> > >> >> > >> On Dec 9, 2013, at 1:46 AM, Anatoly G <anatolyrishon(a)gmail.com> >> wrote: >> > >> >> > >> > With MPICH - 3.0.4 the situation repeated. It looks like MPI >> allocates memory for messages. >> > >> > Can you please advice about scenario when MPI or may be TCP under >> MPI allocates memory due to high transfer rate? >> > >> > >> > >> > >> > >> > On Mon, Dec 9, 2013 at 9:32 AM, Anatoly G <anatolyrishon(a)gmail.com> >> wrote: >> > >> > Thank you very much. >> > >> > Issend - is not so good, It can't support me Fault tolerance. If >> slave process fails, the master stall. >> > >> > I tried mpich-3.0.4 with hydra-3.0.4 but my program which uses MPI >> Fault tolerance doesn't recognize failure of slave process, but recognizes >> failure with MPICH2. May be you can suggest solution? >> > >> > I tried to use hydra from MPICH2 but link my program with MPICH3. >> This combination recognizes failures, but I"m not sure that such >> combination is stable enough. >> > >> > Can you please advice? >> > >> > Anatoly. >> > >> > >> > >> > >> > >> > >> > >> > On Sat, Dec 7, 2013 at 5:20 PM, Pavan Balaji <balaji(a)mcs.anl.gov> >> wrote: >> > >> > >> > >> > As much as I hate saying this — some people find it easier to >> think of it as “MPICH3”. >> > >> > >> > >> > — Pavan >> > >> > >> > >> > On Dec 7, 2013, at 7:37 AM, Wesley Bland <wbland(a)mcs.anl.gov> >> wrote: >> > >> > >> > >> > > MPICH is just the new version of MPICH2. We renamed it when we >> went past version 3.0. >> > >> > > >> > >> > > On Dec 7, 2013, at 3:55 AM, Anatoly G <anatolyrishon(a)gmail.com> >> wrote: >> > >> > > >> > >> > >> Ok. I"ll try both Issend, and next step to upgrade MPICH to >> 3.0.4. >> > >> > >> I thought before that MPICH & MPICH2 are two different >> branches, when MPICH2 partially supports Fault tolerance, but MPICH not. >> Now I understand, that I was wrong and MPICH2 is just main version of MPICH. >> > >> > >> >> > >> > >> Thank you very much, >> > >> > >> Anatoly. >> > >> > >> >> > >> > >> >> > >> > >> >> > >> > >> On Thu, Dec 5, 2013 at 11:01 PM, Rajeev Thakur < >> thakur(a)mcs.anl.gov> wrote: >> > >> > >> The master is receiving too many incoming messages than it can >> match quickly enough with Irecvs. Try using MPI_Issend instead of MPI_Isend. >> > >> > >> >> > >> > >> Rajeev >> > >> > >> >> > >> > >> On Dec 5, 2013, at 2:58 AM, Anatoly G <anatolyrishon(a)gmail.com> >> wrote: >> > >> > >> >> > >> > >> > Hello. >> > >> > >> > I"m using MPICH2 1.5. >> > >> > >> > My system contains master and 16 slaves. >> > >> > >> > System uses number of communicators. >> > >> > >> > The single communicator used for below scenario: >> > >> > >> > Each slave sends non-stop 2Kbyte data buffer using MPI_Isend >> and waits using MPI_Wait. >> > >> > >> > Master starts with MPI_Irecv to each slave >> > >> > >> > Then in endless loop: >> > >> > >> > MPI_Waitany and MPI_Irecv on rank returned by MPI_Waitany. >> > >> > >> > >> > >> > >> > Another communicator used for broadcast communication >> (commands between master + slaves), >> > >> > >> > but it's not used in parallel with previous communicator, >> > >> > >> > only before or after data transfer. >> > >> > >> > >> > >> > >> > The system executed on two computers linked by 1Gbit/s >> Ethernet. >> > >> > >> > Master executed on first computer, all slaves on other one. >> > >> > >> > Network traffic is ~800Mbit/s. >> > >> > >> > >> > >> > >> > After 1-2 minutes of execution, master process starts to >> increase it's memory allocation and network traffic becomes low. >> > >> > >> > This memory allocation & network traffic slow down continues >> until fail of MPI, >> > >> > >> > without core file save. >> > >> > >> > My program doesn't allocate memory. Can you please explain >> this behaviour. >> > >> > >> > How can I cause MPI to stop sending slaves if Master can't >> serve such traffic, instead of memory allocation and fail? >> > >> > >> > >> > >> > >> > >> > >> > >> > Thank you, >> > >> > >> > Anatoly. >> > >> > >> > >> > >> > >> > P.S. >> > >> > >> > On my stand alone test, I simulate similar behaviour, but >> with single thread on each process (master & hosts). >> > >> > >> > When I start stand alone test, master stops slaves until it >> completes accumulated data processing and MPI doesn't increase memory >> allocation. >> > >> > >> > When Master is free slaves continue to send data. >> > >> > >> > _______________________________________________ >> > >> > >> > discuss mailing list discuss(a)mpich.org >> > >> > >> > To manage subscription options or unsubscribe: >> > >> > >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > >> >> > >> > >> _______________________________________________ >> > >> > >> discuss mailing list discuss(a)mpich.org >> > >> > >> To manage subscription options or unsubscribe: >> > >> > >> https://lists.mpich.org/mailman/listinfo/discuss >> > >> > >> >> > >> > >> _______________________________________________ >> > >> > >> discuss mailing list discuss(a)mpich.org >> > >> > >> To manage subscription options or unsubscribe: >> > >> > >> https://lists.mpich.org/mailman/listinfo/discuss >> > >> > > _______________________________________________ >> > >> > > discuss mailing list discuss(a)mpich.org >> > >> > > To manage subscription options or unsubscribe: >> > >> > > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > >> > >> > -- >> > >> > Pavan Balaji >> > >> > http://www.mcs.anl.gov/~balaji >> > >> > >> > >> > _______________________________________________ >> > >> > discuss mailing list discuss(a)mpich.org >> > >> > To manage subscription options or unsubscribe: >> > >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > >> > >> > >> > >> > _______________________________________________ >> > >> > discuss mailing list discuss(a)mpich.org >> > >> > To manage subscription options or unsubscribe: >> > >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> >> > >> -- >> > >> Pavan Balaji >> > >> http://www.mcs.anl.gov/~balaji >> > >> >> > >> _______________________________________________ >> > >> discuss mailing list discuss(a)mpich.org >> > >> To manage subscription options or unsubscribe: >> > >> https://lists.mpich.org/mailman/listinfo/discuss >> > >> >> > >> >> > >> _______________________________________________ >> > >> discuss mailing list discuss(a)mpich.org >> > >> To manage subscription options or unsubscribe: >> > >> https://lists.mpich.org/mailman/listinfo/discuss >> > > >> > > _______________________________________________ >> > > discuss mailing list discuss(a)mpich.org >> > > To manage subscription options or unsubscribe: >> > > https://lists.mpich.org/mailman/listinfo/discuss >> > > >> > > _______________________________________________ >> > > discuss mailing list discuss(a)mpich.org >> > > To manage subscription options or unsubscribe: >> > > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > -- >> > Pavan Balaji >> > http://www.mcs.anl.gov/~balaji >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > >

1 0