April 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH (v3.1) Performing slow on 10G Ethernet
by Atchley, Scott 16 Apr '14

16 Apr '14

On Apr 16, 2014, at 8:35 AM, Scott Atchley <atchleyes(a)ornl.gov> wrote: > Keep in mind that iperf is a streaming test. What MPI application are you running? IMB pingpong? > > You should review you NIC vendor's tuning FAQ. Is interrupt coalescing turned on? Normally it is, but it increases small message latency. > > As for bandwidth, have you tried a streaming MPI benchmark? Or run Netperf's TCP_RR for each message size to see how it compares to pingpong. > > Scott > > On Apr 15, 2014, at 4:29 PM, Muhammad Ansar Javed <muhammad.ansar(a)seecs.edu.pk> wrote: > >> Hi Antonio, >> Thanks for response. >> Here is output of iperf bandwidth test. >> . >> Client connecting to 10g2, TCP port 5001 >> TCP window size: 1.41 MByte (default) >> ------------------------------------------------------------ >> [ 5] local 192.168.1.33 port 34543 connected with 192.168.1.34 port 5001 >> [ 4] local 192.168.1.33 port 5001 connected with 192.168.1.34 port 48531 >> [ ID] Interval Transfer Bandwidth >> [ 5] 0.0-10.0 sec 11.2 GBytes 9.61 Gbits/sec >> [ 4] 0.0-10.0 sec 11.2 GBytes 9.59 Gbits/sec >> >> >> >> On Tue, Apr 15, 2014 at 7:17 PM, "Antonio J. Peña" <apenya(a)mcs.anl.gov> wrote: >> >> Hi Ansar, >> >> The bandwidth numbers seem a little low. I'd expect something around 7,000 Mbps. To discard issues with MPI, you can try generic benchmarking tools such as those described here> >> >> http://linuxaria.com/article/tool-command-line-bandwidth-linux?lang=en >> >> If you get similar numbers, you'll have to check your setup. Otherwise, let us know and we'll try to see if there's any performance issue with MPICH. >> >> Best, >> Antonio >> >> >> >> On 04/15/2014 06:35 AM, Muhammad Ansar Javed wrote: >>> Hi, >>> I am running benchmarks for MPICH v3.1 performance evaluation on 10G Ethernet connection between two hosts. The performance results are less than expected. Here is complete set of numbers for Latency and Bandwidth tests. >>> >>> mpj@host3:~/code/benchmarks$ mpiexec -n 2 -f machines ./latency.out >>> Latency: MPICH-C >>> Size ______ Time (us) >>> 1 _________ 37.91 >>> 2 _________ 37.76 >>> 4 _________ 37.75 >>> 8 _________ 39.02 >>> 16 ________ 52.71 >>> 32 ________ 39.08 >>> 64 ________ 37.75 >>> 128 _______ 37.77 >>> 256 _______ 57.47 >>> 512 _______ 37.86 >>> 1024 ______ 37.76 >>> 2048 ______ 37.88 >>> >>> mpj@host3:~/code/benchmarks$ mpiexec -n 2 -f machines ./bandwidth.out >>> Bandwidth: MPICH-C >>> Size(Bytes) Bandwidth (Mbps) >>> 2048 ______ 412.32 >>> 4096 ______ 820.06 >>> 8192 ______ 827.77 >>> 16384 _____ 1644.36 >>> 32768 _____ 2207.52 >>> 65536 _____ 4368.76 >>> 131072 ____ 2942.93 >>> 262144 ____ 4281.17 >>> 524288 ____ 4773.78 >>> 1048576 ___ 5310.85 >>> 2097152 ___ 5382.94 >>> 4194304 ___ 5518.97 >>> 8388608 ___ 5508.87 >>> 16777216 __ 5498.93 >>> >>> My environments consists of two hosts having point-to-point (switch-less) 10Gbps Ethernet connection. Environment (OS, user, directory structure etc) on both hosts is exactly same. There is no NAS or shared file system between hosts. I have attached output of script >>> mpiexec -all. >>> >>> Are these numbers okay? If not then please suggest performance improvement methodology... >>> >>> Thanks >>> >>> -- >>> Regards >>> >>> Ansar Javed >>> HPC Lab >>> SEECS NUST >>> Contact: +92 334 438 9394 >>> Email: muhammad.ansar(a)seecs.edu.pk >>> >>> >>> _______________________________________________ >>> discuss mailing list >>> discuss(a)mpich.org >>> >>> To manage subscription options or unsubscribe: >>> >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> -- >> Antonio J. Peña >> Postdoctoral Appointee >> Mathematics and Computer Science Division >> Argonne National Laboratory >> 9700 South Cass Avenue, Bldg. 240, Of. 3148 >> Argonne, IL 60439-4847 >> >> apenya(a)mcs.anl.gov >> www.mcs.anl.gov/~apenya >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> -- >> Regards >> >> Ansar Javed >> HPC Lab >> SEECS NUST >> Contact: +92 334 438 9394 >> Email: muhammad.ansar(a)seecs.edu.pk >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPICH (v3.1) Performing slow on 10G Ethernet
by Atchley, Scott 16 Apr '14

16 Apr '14

Keep in mind that iperf is a streaming test. What MPI application are you running? IMB pingpong? You should review you NIC vendor's tuning FAQ. Is interrupt coalescing turned on? Normally it is, but it increases small message latency. As for bandwidth, have you tried a streaming MPI benchmark? Scott On Apr 15, 2014, at 4:29 PM, Muhammad Ansar Javed <muhammad.ansar(a)seecs.edu.pk> wrote: > Hi Antonio, > Thanks for response. > Here is output of iperf bandwidth test. > . > Client connecting to 10g2, TCP port 5001 > TCP window size: 1.41 MByte (default) > ------------------------------------------------------------ > [ 5] local 192.168.1.33 port 34543 connected with 192.168.1.34 port 5001 > [ 4] local 192.168.1.33 port 5001 connected with 192.168.1.34 port 48531 > [ ID] Interval Transfer Bandwidth > [ 5] 0.0-10.0 sec 11.2 GBytes 9.61 Gbits/sec > [ 4] 0.0-10.0 sec 11.2 GBytes 9.59 Gbits/sec > > > > On Tue, Apr 15, 2014 at 7:17 PM, "Antonio J. Peña" <apenya(a)mcs.anl.gov> wrote: > > Hi Ansar, > > The bandwidth numbers seem a little low. I'd expect something around 7,000 Mbps. To discard issues with MPI, you can try generic benchmarking tools such as those described here> > > http://linuxaria.com/article/tool-command-line-bandwidth-linux?lang=en > > If you get similar numbers, you'll have to check your setup. Otherwise, let us know and we'll try to see if there's any performance issue with MPICH. > > Best, > Antonio > > > > On 04/15/2014 06:35 AM, Muhammad Ansar Javed wrote: >> Hi, >> I am running benchmarks for MPICH v3.1 performance evaluation on 10G Ethernet connection between two hosts. The performance results are less than expected. Here is complete set of numbers for Latency and Bandwidth tests. >> >> mpj@host3:~/code/benchmarks$ mpiexec -n 2 -f machines ./latency.out >> Latency: MPICH-C >> Size ______ Time (us) >> 1 _________ 37.91 >> 2 _________ 37.76 >> 4 _________ 37.75 >> 8 _________ 39.02 >> 16 ________ 52.71 >> 32 ________ 39.08 >> 64 ________ 37.75 >> 128 _______ 37.77 >> 256 _______ 57.47 >> 512 _______ 37.86 >> 1024 ______ 37.76 >> 2048 ______ 37.88 >> >> mpj@host3:~/code/benchmarks$ mpiexec -n 2 -f machines ./bandwidth.out >> Bandwidth: MPICH-C >> Size(Bytes) Bandwidth (Mbps) >> 2048 ______ 412.32 >> 4096 ______ 820.06 >> 8192 ______ 827.77 >> 16384 _____ 1644.36 >> 32768 _____ 2207.52 >> 65536 _____ 4368.76 >> 131072 ____ 2942.93 >> 262144 ____ 4281.17 >> 524288 ____ 4773.78 >> 1048576 ___ 5310.85 >> 2097152 ___ 5382.94 >> 4194304 ___ 5518.97 >> 8388608 ___ 5508.87 >> 16777216 __ 5498.93 >> >> My environments consists of two hosts having point-to-point (switch-less) 10Gbps Ethernet connection. Environment (OS, user, directory structure etc) on both hosts is exactly same. There is no NAS or shared file system between hosts. I have attached output of script >> mpiexec -all. >> >> Are these numbers okay? If not then please suggest performance improvement methodology... >> >> Thanks >> >> -- >> Regards >> >> Ansar Javed >> HPC Lab >> SEECS NUST >> Contact: +92 334 438 9394 >> Email: muhammad.ansar(a)seecs.edu.pk >> >> >> _______________________________________________ >> discuss mailing list >> discuss(a)mpich.org >> >> To manage subscription options or unsubscribe: >> >> https://lists.mpich.org/mailman/listinfo/discuss > > > -- > Antonio J. Peña > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847 > > apenya(a)mcs.anl.gov > www.mcs.anl.gov/~apenya > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Regards > > Ansar Javed > HPC Lab > SEECS NUST > Contact: +92 334 438 9394 > Email: muhammad.ansar(a)seecs.edu.pk > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH (v3.1) Performing slow on 10G Ethernet
by Muhammad Ansar Javed 15 Apr '14

15 Apr '14

Hi Antonio, Thanks for response. Here is output of iperf bandwidth test. . Client connecting to 10g2, TCP port 5001 TCP window size: 1.41 MByte (default) ------------------------------------------------------------ [ 5] local 192.168.1.33 port 34543 connected with 192.168.1.34 port 5001 [ 4] local 192.168.1.33 port 5001 connected with 192.168.1.34 port 48531 [ ID] Interval Transfer Bandwidth [ 5] 0.0-10.0 sec 11.2 GBytes 9.61 Gbits/sec [ 4] 0.0-10.0 sec 11.2 GBytes 9.59 Gbits/sec On Tue, Apr 15, 2014 at 7:17 PM, "Antonio J. Peña" <apenya(a)mcs.anl.gov>wrote: > > Hi Ansar, > > The bandwidth numbers seem a little low. I'd expect something around 7,000 > Mbps. To discard issues with MPI, you can try generic benchmarking tools > such as those described here> > > http://linuxaria.com/article/tool-command-line-bandwidth-linux?lang=en > > If you get similar numbers, you'll have to check your setup. Otherwise, > let us know and we'll try to see if there's any performance issue with > MPICH. > > Best, > Antonio > > > > On 04/15/2014 06:35 AM, Muhammad Ansar Javed wrote: > > Hi, > I am running benchmarks for MPICH v3.1 performance evaluation on 10G > Ethernet connection between two hosts. The performance results are less > than expected. Here is complete set of numbers for Latency and Bandwidth > tests. > > mpj@host3:~/code/benchmarks$ mpiexec -n 2 -f machines ./latency.out > Latency: MPICH-C > Size ______ Time (us) > 1 _________ 37.91 > 2 _________ 37.76 > 4 _________ 37.75 > 8 _________ 39.02 > 16 ________ 52.71 > 32 ________ 39.08 > 64 ________ 37.75 > 128 _______ 37.77 > 256 _______ 57.47 > 512 _______ 37.86 > 1024 ______ 37.76 > 2048 ______ 37.88 > > mpj@host3:~/code/benchmarks$ mpiexec -n 2 -f machines ./bandwidth.out > Bandwidth: MPICH-C > Size(Bytes) Bandwidth (Mbps) > 2048 ______ 412.32 > 4096 ______ 820.06 > 8192 ______ 827.77 > 16384 _____ 1644.36 > 32768 _____ 2207.52 > 65536 _____ 4368.76 > 131072 ____ 2942.93 > 262144 ____ 4281.17 > 524288 ____ 4773.78 > 1048576 ___ 5310.85 > 2097152 ___ 5382.94 > 4194304 ___ 5518.97 > 8388608 ___ 5508.87 > 16777216 __ 5498.93 > > My environments consists of two hosts having point-to-point (switch-less) > 10Gbps Ethernet connection. Environment (OS, user, directory structure > etc) on both hosts is exactly same. There is no NAS or shared file system > between hosts. I have attached output of script > mpiexec -all. > > Are these numbers okay? If not then please suggest performance improvement > methodology... > > Thanks > > -- > Regards > > Ansar Javed > HPC Lab > SEECS NUST > Contact: +92 334 438 9394 > Email: muhammad.ansar(a)seecs.edu.pk > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe:https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Antonio J. Peña > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847apenya(a)mcs.anl.govwww.mcs.anl.gov/~apenya > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Regards Ansar Javed HPC Lab SEECS NUST Contact: +92 334 438 9394 Email: muhammad.ansar(a)seecs.edu.pk

1 0

Re: [mpich-discuss] MPICH (v3.1) Performing slow on 10G Ethernet
by "Antonio J. Peña" 15 Apr '14

15 Apr '14

Hi Ansar, The bandwidth numbers seem a little low. I'd expect something around 7,000 Mbps. To discard issues with MPI, you can try generic benchmarking tools such as those described here> http://linuxaria.com/article/tool-command-line-bandwidth-linux?lang=en If you get similar numbers, you'll have to check your setup. Otherwise, let us know and we'll try to see if there's any performance issue with MPICH. Best, Antonio On 04/15/2014 06:35 AM, Muhammad Ansar Javed wrote: > Hi, > I am running benchmarks for MPICH v3.1 performance evaluation on 10G > Ethernet connection between two hosts. The performance results are > less than expected. Here is complete set of numbers for Latency and > Bandwidth tests. > > mpj@host3:~/code/benchmarks$ mpiexec -n 2 -f machines ./latency.out > Latency: MPICH-C > Size ______ Time (us) > 1 _________ 37.91 > 2 _________ 37.76 > 4 _________ 37.75 > 8 _________ 39.02 > 16 ________ 52.71 > 32 ________ 39.08 > 64 ________ 37.75 > 128 _______ 37.77 > 256 _______ 57.47 > 512 _______ 37.86 > 1024 ______ 37.76 > 2048 ______ 37.88 > > mpj@host3:~/code/benchmarks$ mpiexec -n 2 -f machines ./bandwidth.out > Bandwidth: MPICH-C > Size(Bytes) Bandwidth (Mbps) > 2048 ______ 412.32 > 4096 ______ 820.06 > 8192 ______ 827.77 > 16384 _____ 1644.36 > 32768 _____ 2207.52 > 65536 _____ 4368.76 > 131072 ____ 2942.93 > 262144 ____ 4281.17 > 524288 ____ 4773.78 > 1048576 ___ 5310.85 > 2097152 ___ 5382.94 > 4194304 ___ 5518.97 > 8388608 ___ 5508.87 > 16777216 __ 5498.93 > > My environments consists of two hosts having point-to-point > (switch-less) 10Gbps Ethernet connection. Environment (OS, user, > directory structure etc) on both hosts is exactly same. There is no > NAS or shared file system between hosts. I have attached output of script > mpiexec -all. > > Are these numbers okay? If not then please suggest performance > improvement methodology... > > Thanks > > -- > Regards > > Ansar Javed > HPC Lab > SEECS NUST > Contact: +92 334 438 9394 > Email: muhammad.ansar(a)seecs.edu.pk <mailto:[email protected]> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peï¿½a Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya

1 0

MPICH (v3.1) Performing slow on 10G Ethernet
by Muhammad Ansar Javed 15 Apr '14

15 Apr '14

Hi, I am running benchmarks for MPICH v3.1 performance evaluation on 10G Ethernet connection between two hosts. The performance results are less than expected. Here is complete set of numbers for Latency and Bandwidth tests. mpj@host3:~/code/benchmarks$ mpiexec -n 2 -f machines ./latency.out Latency: MPICH-C Size ______ Time (us) 1 _________ 37.91 2 _________ 37.76 4 _________ 37.75 8 _________ 39.02 16 ________ 52.71 32 ________ 39.08 64 ________ 37.75 128 _______ 37.77 256 _______ 57.47 512 _______ 37.86 1024 ______ 37.76 2048 ______ 37.88 mpj@host3:~/code/benchmarks$ mpiexec -n 2 -f machines ./bandwidth.out Bandwidth: MPICH-C Size(Bytes) Bandwidth (Mbps) 2048 ______ 412.32 4096 ______ 820.06 8192 ______ 827.77 16384 _____ 1644.36 32768 _____ 2207.52 65536 _____ 4368.76 131072 ____ 2942.93 262144 ____ 4281.17 524288 ____ 4773.78 1048576 ___ 5310.85 2097152 ___ 5382.94 4194304 ___ 5518.97 8388608 ___ 5508.87 16777216 __ 5498.93 My environments consists of two hosts having point-to-point (switch-less) 10Gbps Ethernet connection. Environment (OS, user, directory structure etc) on both hosts is exactly same. There is no NAS or shared file system between hosts. I have attached output of script mpiexec -all. Are these numbers okay? If not then please suggest performance improvement methodology... Thanks -- Regards Ansar Javed HPC Lab SEECS NUST Contact: +92 334 438 9394 Email: muhammad.ansar(a)seecs.edu.pk

1 0

Re: [mpich-discuss] my mpi code goes slow after some time
by Lu, Huiwei 14 Apr '14

14 Apr '14

(Please don’t drop the discussion thread, added it back.) Is 1.0 second for one step or for the whole application time? Also, can you provide the following information of your program? 1. What’s the version of your MPI? 2. How many processes you use? 3. Can you give us a minimum working example of your code so that we can reproduce your problem? You can read a tutorial of parallel computing here: https://computing.llnl.gov/tutorials/parallel_comp/ — Huiwei On Apr 14, 2014, at 10:54 AM, harsh soni <harshiisc(a)gmail.com> wrote: > Dear Huiwei, > > Thanks for the replay. I tried to calculate the time taken by the various segments in my code. I realize, the slowest part correspond to hug communication between cores, with communication of 15750 double precision pre core, at every step. The time taken by this communication is around 1.0 second. I just want to know, how the communication affects the speed of code. I read about "parallel slowdown". Can please tell me something about it. > > Sorry for bothering you! > > Thanks > Harsh Soni > Ph.D. student, > IISc, Bengaluru (India) > > > > On Thu, Apr 10, 2014 at 6:47 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> wrote: > How slow is that? What’s the version of your MPI? How many processes you use? > > If only 0.3% RAM was used for each core, it was unlikely the code went slow because of running out of RAM. Can you give us a minimum working example of your code so that we can reproduce your problem? > > — > Huiwei Lu > Postdoc Appointee > Mathematics and Computer Science Division, Argonne National Laboratory > http://www.mcs.anl.gov/~huiweilu/ > > On Apr 10, 2014, at 7:57 AM, harsh soni <harshiisc(a)gmail.com> wrote: > > > Hi, > > I am running a mpi code on cluster. It goes slow after some time. I don't allocate any array in main for loop, every array is allocated initially. Each core is showing 0.3 to 0.5 % RAM usage. doe sit go slow due to the RAM. I use fortran language. > > > > > > Thanks > > > > With best regards > > Harsh Soni > > Ph.D. student, > > IISc, Bengaluru (India) > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] problem with mpiexec
by Gus Correa 11 Apr '14

11 Apr '14

In addition to Gilles suggestion, please check the answers to similar questions you posted a few days ago about the machinefile/hostfile syntax. They are archived here: http://lists.mpich.org/pipermail/discuss/2014-April/002486.html This error message suggests that it doesn't exist, is misplaced, or is not readable: mpiexec@Mojtaba] HYDU_parse_hostfile (utils/args/args.c:319): unable to open host file: ./machinefile If you preceed the machinefile name by "./" it should be in the same directory where you launch the mpiexec command line. Make sure the file exists, follows the standard syntax, and has read permissions open. On 04/11/2014 06:59 AM, Gilles Gouaillardet wrote: > Hi, > > you need to create the machinefile manually > this is a text file, and the basic syntax is one hostname per line > > as i wrote earlier, i recommend you use > -host localhost > instead of > -machinefile machinefile > > Cheers, > > Gilles > > On 2014/04/11 19:56, Mojtaba S. Moghani wrote: >> >> >> Hi >> >> I cant see the file named machinefile, where it should be? >> >> and if i dont have that, how i should make that file? >> >> how i could know that im done with mpich-3.1 installation? >> >> thanks, >> >> >> >> _______________________________________________ >> discuss mailing listdiscuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] problem with mpiexec
by Gilles Gouaillardet 11 Apr '14

11 Apr '14

Hi, you need to create the machinefile manually this is a text file, and the basic syntax is one hostname per line as i wrote earlier, i recommend you use -host localhost instead of -machinefile machinefile Cheers, Gilles On 2014/04/11 19:56, Mojtaba S. Moghani wrote: > > > Hi > > I cant see the file named machinefile, where it should be? > > and if i dont have that, how i should make that file? > > how i could know that im done with mpich-3.1 installation? > > thanks, > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

problem with mpiexec
by Mojtaba S. Moghani 11 Apr '14

11 Apr '14

Hi I cant see the file named machinefile, where it should be? and if i dont have that, how i should make that file? how i could know that im done with mpich-3.1 installation? thanks, -- Mojtaba S. Moghani Faculty of Environment University of Tehran

1 0

Re: [mpich-discuss] problem with mpiexec
by Mojtaba S. Moghani 11 Apr '14

11 Apr '14

I cant see the file named machinefile, where it should be? and if i dont have that, how i should make that file? how i could know that im done with mpich-3.1 installation? thanks, On 2014-04-11 13:31, Gilles Gouaillardet wrote: > Hi, > > did you create a file called machinefile ? > > if yes, can you send its content ? > > an easier way to start with mpiexec is : > mpiexec -n 5 -host localhost ./examples/cpi > > Cheers, > > Gilles > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss [1] -- Mojtaba S. Moghani Faculty of Environment University of Tehran Links: ------ [1] https://lists.mpich.org/mailman/listinfo/discuss

1 0