April 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Isend and Test
by Rob Latham 25 Apr '16

25 Apr '16

On 04/25/2016 02:57 PM, Zhen Wang wrote: > Hi, > > I have questions regarding MPI_Isend and MPI_Test. A sample code is > attached, output is as follows. > > For instance, both Isend and Recv of 0 start at 15:46:14, but 2 MPI_Test > are called before Recv of 0 finishes. My understanding is first Test > receives the signal that receiver is ready, and data transfer follows. > Then the second Test will see the transfer is complete, and free > MPI_Request. Is my understanding correct? Thanks. MPI_Test() is non-blocking. MPI_Test() is not going to wait for anything to happen -- that's what MPI_Wait() will do. under the hood, MPI_Test "kicks the progress engine", which means everything that can execute right now will happen. If there is anything that requires code to wait (to receive a signal, in your case), then test will return. A data transfer might take several calls to MPI_Test to complete. Or, there's a background progress thread that's churning along and you only need to make one call to MPI_Test. It's implementation-dependent. You are asking the wrong question, though. At this level, it doesn't matter what MPI_Test does. If your code has something it can do while the isend progresses, then go do that and call MPI_Test periodically. If you have nothing productive you can do, call MPI_Wait() (or one of the variants). ==rob > > > Best regards, > Zhen > > MPI 1: Recv of 0 started at 15:46:14. > MPI 0: Isend of 0 started at 15:46:14. > MPI 0: Isend of 1 started at 15:46:14. > MPI 0: Isend of 2 started at 15:46:14. > MPI 0: Isend of 3 started at 15:46:14. > MPI 0: Isend of 4 started at 15:46:14. > MPI 0: MPI_Test of 0 at 15:46:16. > MPI 0: MPI_Test of 0 at 15:46:18. > MPI 0: Isend of 0 finished at 15:46:18. > MPI 1: Recv of 0 finished at 15:46:18. > MPI 1: Recv of 1 started at 15:46:18. > MPI 0: MPI_Test of 1 at 15:46:20. > MPI 0: MPI_Test of 1 at 15:46:22. > MPI 0: Isend of 1 finished at 15:46:22. > MPI 1: Recv of 1 finished at 15:46:22. > MPI 1: Recv of 2 started at 15:46:22. > MPI 0: MPI_Test of 2 at 15:46:24. > MPI 0: MPI_Test of 2 at 15:46:26. > MPI 0: Isend of 2 finished at 15:46:26. > MPI 1: Recv of 2 finished at 15:46:26. > MPI 1: Recv of 3 started at 15:46:26. > MPI 0: MPI_Test of 3 at 15:46:28. > MPI 0: MPI_Test of 3 at 15:46:30. > MPI 0: Isend of 3 finished at 15:46:30. > MPI 1: Recv of 3 finished at 15:46:30. > MPI 1: Recv of 4 started at 15:46:30. > MPI 0: MPI_Test of 4 at 15:46:32. > MPI 0: MPI_Test of 4 at 15:46:34. > MPI 0: Isend of 4 finished at 15:46:34. > MPI 1: Recv of 4 finished at 15:46:34. > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Isend and Test
by Zhen Wang 25 Apr '16

25 Apr '16

Hi, I have questions regarding MPI_Isend and MPI_Test. A sample code is attached, output is as follows. For instance, both Isend and Recv of 0 start at 15:46:14, but 2 MPI_Test are called before Recv of 0 finishes. My understanding is first Test receives the signal that receiver is ready, and data transfer follows. Then the second Test will see the transfer is complete, and free MPI_Request. Is my understanding correct? Thanks. Best regards, Zhen MPI 1: Recv of 0 started at 15:46:14. MPI 0: Isend of 0 started at 15:46:14. MPI 0: Isend of 1 started at 15:46:14. MPI 0: Isend of 2 started at 15:46:14. MPI 0: Isend of 3 started at 15:46:14. MPI 0: Isend of 4 started at 15:46:14. MPI 0: MPI_Test of 0 at 15:46:16. MPI 0: MPI_Test of 0 at 15:46:18. MPI 0: Isend of 0 finished at 15:46:18. MPI 1: Recv of 0 finished at 15:46:18. MPI 1: Recv of 1 started at 15:46:18. MPI 0: MPI_Test of 1 at 15:46:20. MPI 0: MPI_Test of 1 at 15:46:22. MPI 0: Isend of 1 finished at 15:46:22. MPI 1: Recv of 1 finished at 15:46:22. MPI 1: Recv of 2 started at 15:46:22. MPI 0: MPI_Test of 2 at 15:46:24. MPI 0: MPI_Test of 2 at 15:46:26. MPI 0: Isend of 2 finished at 15:46:26. MPI 1: Recv of 2 finished at 15:46:26. MPI 1: Recv of 3 started at 15:46:26. MPI 0: MPI_Test of 3 at 15:46:28. MPI 0: MPI_Test of 3 at 15:46:30. MPI 0: Isend of 3 finished at 15:46:30. MPI 1: Recv of 3 finished at 15:46:30. MPI 1: Recv of 4 started at 15:46:30. MPI 0: MPI_Test of 4 at 15:46:32. MPI 0: MPI_Test of 4 at 15:46:34. MPI 0: Isend of 4 finished at 15:46:34. MPI 1: Recv of 4 finished at 15:46:34. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH2-1.4.1 on Windows Server, Issues Running on More than One Node
by Joshua Moore 24 Apr '16

24 Apr '16

Hello, I am having issues running mpiexec when I run on more than one node. I am using v 1.4.1 because it is compatible with LAMMPS software I am using. I am running on Windows Server 2012r2 and trying to get MPICH2 to play nice with Window's batch server. I've installed mpich2 on each of the nodes and the head node. 1) *msiexec /i mpich2-1.2.1-win-ia32.msi* *2) mpiexec -register (to register username and password)* *3) smpd -install (to start spmd server on **each of the nodes and the head node)* *4) On the head node, I've used smpd -sethosts hostname1 hostname2 ... to add each of the hosts. I haven't done this on the compute nodes.* When I execute a job through Windows batch system through PowerShell, my job executes but puts all of the executions on the same node. So if for example my nodes are 16 cores, and I ask for 32 cores, 32 separate processes are run on the first node and none on the second, even though Window's scheduler is allocated 2 nodes. It's like mpiexec is ignoring the node list that the smpd hosts is setting. I can use a machinefile with mpiexec and this will allow me to execute on multiple nodes but it doesn't seem to follow the nodes that Windows allocates in their batch server. I should also add that when I try to request more than one node through Window's batch server with -requestednodes "host1 host2" with new-hpcjob, Windows tells me that I can't do this because I have zero cores available. I can ask for only 1 node and up to the maximum number of cores and it is ok with this. Any suggestions? Thank you. Josh _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

runtime error for mpich-master-v3.2-247-g1aec69b70951 with Solaris Sparc
by Siegmar Gross 21 Apr '16

21 Apr '16

Hi, I have built mpich-master-v3.2-247-g1aec69b70951 on my machines (Solaris 10 Sparc, Solaris 10 x86_64, and openSUSE Linux 12.1 x86_64) with gcc-5.1.0 and Sun C 5.13. I get the following errors for both compilers if I run small programs that spawn processes on two Sparc machines. Everything works fine if I use Linux and/or Solaris x86_64. "mpiexec" is aliased to 'mpiexec -genvnone'. I get different errors, if I run the same command several times as you can see below (sometimes it even works as expected). tyr spawn 119 mpichversion MPICH Version: 3.2 MPICH Release date: Tue Apr 19 00:00:44 CDT 2016 MPICH Device: ch3:nemesis MPICH configure: --prefix=/usr/local/mpich-3.2.1_64_gcc --libdir=/usr/local/mpich-3.2.1_64_gcc/lib64 --includedir=/usr/local/mpich-3.2.1_64_gcc/include64 CC=gcc CXX=g++ F77=gfortran FC=gfortran CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 FCFLAGS=-m64 LDFLAGS=-m64 -L/usr/lib/sparcv9 -Wl,-rpath -Wl,/usr/lib/sparcv9 --enable-fortran=yes --enable-cxx --enable-romio --enable-debuginfo --enable-smpcoll --enable-threads=multiple --with-thread-package=posix --enable-shared MPICH CC: gcc -m64 -O2 MPICH CXX: g++ -m64 -O2 MPICH F77: gfortran -m64 -O2 MPICH FC: gfortran -m64 -O2 tyr spawn 120 mpiexec -np 1 --host tyr,tyr,tyr,ruester,ruester spawn_master Parent process 0 running on tyr.informatik.hs-fulda.de I create 4 slave processes Fatal error in MPI_Comm_spawn: Unknown error class, error stack: MPI_Comm_spawn(144)...................: MPI_Comm_spawn(cmd="spawn_slave", argv=0, maxprocs=4, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, intercomm=ffffffff7fffdf58, errors=0) failed MPIDI_Comm_spawn_multiple(274)........: MPID_Comm_accept(153).................: MPIDI_Comm_accept(1039)...............: MPIDU_Complete_posted_with_error(1137): Process failed =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 3182 RUNNING AT tyr = EXIT CODE: 10 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== tyr spawn 121 mpiexec -np 1 --host tyr,tyr,tyr,ruester,ruester spawn_master Parent process 0 running on tyr.informatik.hs-fulda.de I create 4 slave processes Parent process 0: tasks in MPI_COMM_WORLD: 1 tasks in COMM_CHILD_PROCESSES local group: 1 tasks in COMM_CHILD_PROCESSES remote group: 4 Slave process 3 of 4 running on ruester.informatik.hs-fulda.de Slave process 2 of 4 running on ruester.informatik.hs-fulda.de spawn_slave 2: argv[0]: spawn_slave spawn_slave 3: argv[0]: spawn_slave Slave process 0 of 4 running on tyr.informatik.hs-fulda.de spawn_slave 0: argv[0]: spawn_slave Slave process 1 of 4 running on tyr.informatik.hs-fulda.de spawn_slave 1: argv[0]: spawn_slave tyr spawn 122 mpiexec -np 1 --host tyr,tyr,tyr,ruester,ruester spawn_master Parent process 0 running on tyr.informatik.hs-fulda.de I create 4 slave processes tyr spawn 123 mpiexec -np 1 --host tyr,tyr,tyr,ruester,ruester spawn_master Parent process 0 running on tyr.informatik.hs-fulda.de I create 4 slave processes Fatal error in MPI_Comm_spawn: Unknown error class, error stack: MPI_Comm_spawn(144)...................: MPI_Comm_spawn(cmd="spawn_slave", argv=0, maxprocs=4, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, intercomm=ffffffff7fffdf58, errors=0) failed MPIDI_Comm_spawn_multiple(274)........: MPID_Comm_accept(153).................: MPIDI_Comm_accept(1039)...............: MPIDU_Complete_posted_with_error(1137): Process failed tyr spawn 124 mpiexec -np 1 --host tyr,tyr,tyr,ruester,ruester spawn_master Parent process 0 running on tyr.informatik.hs-fulda.de I create 4 slave processes Fatal error in MPI_Comm_spawn: Unknown error class, error stack: MPI_Comm_spawn(144)...................: MPI_Comm_spawn(cmd="spawn_slave", argv=0, maxprocs=4, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, intercomm=ffffffff7fffdf58, errors=0) failed MPIDI_Comm_spawn_multiple(274)........: MPID_Comm_accept(153).................: MPIDI_Comm_accept(1039)...............: MPIDU_Complete_posted_with_error(1137): Process failed =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 3466 RUNNING AT tyr = EXIT CODE: 10 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-master-v3.2-247-g1aec69b70951/src/pm/hydra/pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:0:[email protected][proxy:1:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-master-v3.2-247-g1aec69b70951/src/pm/hydra/pm/pmiserv/pmip_cb.c] HYDT_dmxu_poll_wait_for_event (../../../../mpich-master-v3.2-247-g1aec69b70951/src/pm/hydra/tools/demux/demux_poll.c:77): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-master-v3.2-247-g1aec69b70951/src/pm/hydra/pm/pmiserv/pmip.c:202): demux engine error waiting for event [mpiexec(a)tyr.informatik.hs-fulda.de] HYDT_bscu_wait_for_completion (../../../../mpich-master-v3.2-247-g1aec69b70951/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec(a)tyr.informatik.hs-fulda.de] HYDT_bsci_wait_for_completion (../../../../mpich-master-v3.2-247-g1aec69b70951/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec(a)tyr.informatik.hs-fulda.de] HYD_pmci_wait_for_completion (../../../../mpich-master-v3.2-247-g1aec69b70951/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec(a)tyr.informatik.hs-fulda.de] main (../../../../mpich-master-v3.2-247-g1aec69b70951/src/pm/hydra/ui/mpich/mpiexec.c:340): process manager error waiting for completion tyr spawn 125 I would be grateful if somebody can fix the problem. Thank you very much for any help in advance. Kind regards Siegmar _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] collective read slower than non-collective read
by 冰 21 Apr '16

21 Apr '16

Hi rob, I look at the src/mpi/romio/test/coll_perf.c test, and I experimented with the collective and non-collective versions, in single machine collective read is slower than non-collective read, ,my example is behind. #include <stdio.h> #include <stdlib.h> #include "mpi.h" #define FILE_LENGTH 100000000 #define UNSIGNEDCHAR_PER_BLK 10 int main( int argc, char **argv ) { int myid, numprocs; double startTime,readTime,newReadTime; unsigned char* buf; int bufLen; MPI_Datatype fileType; MPI_File fh; MPI_Status status; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD,&myid); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); bufLen=FILE_LENGTH/numprocs; buf=(unsigned char*) malloc(bufLen); MPI_File_open(MPI_COMM_WORLD,"g://test.txt",MPI_MODE_RDWR,MPI_INFO_NULL,&fh); MPI_Type_vector(bufLen/UNSIGNEDCHAR_PER_BLK,UNSIGNEDCHAR_PER_BLK,numprocs*UNSIGNEDCHAR_PER_BLK,MPI_UNSIGNED_CHAR,&fileType); MPI_Type_commit(&fileType); MPI_File_set_view(fh, myid*UNSIGNEDCHAR_PER_BLK, MPI_UNSIGNED_CHAR, fileType, "native", MPI_INFO_NULL); startTime=MPI_Wtime(); //non-collective read //MPI_File_read(fh,buf,bufLen,MPI_UNSIGNED_CHAR,&status); //collective read MPI_File_read_all(fh,buf,bufLen,MPI_UNSIGNED_CHAR,&status); readTime=MPI_Wtime()-startTime; MPI_Allreduce(&readTime, &newReadTime, 1, MPI_DOUBLE, MPI_MAX,MPI_COMM_WORLD); if (myid == 0) { printf("%d processes read time = %f sec, read bandwidth = %f Mbytes/sec\n",numprocs, newReadTime, FILE_LENGTH/(1024*1024*newReadTime)); } free(buf); MPI_File_close(&fh); MPI_Finalize(); } the result is non-collective read 4 processes read time = 1.380783 sec, read bandwidth = 69.067633 Mbytes/sec collective read 4 processes read time = 2.256306 sec, read bandwidth = 42.267062 Mbytes/sec I am interested in collective read/write, mybe I should experiment collective I/O in cluster, thank you for your answer. Best Regard, Ice. At 2016-04-20 01:49:08, "Rob Latham" <robl(a)mcs.anl.gov> wrote: > > >On 04/19/2016 07:45 AM, 冰 wrote: >> Hi, >> I run programs in one computer, In my tests here, the MPI_File_read_all >> have been slower than MPI_File_read, could you send me a simple routine >> that collective read is faster than non-collective read? > >Need more information. > >The collective calls provide two important optimizations: reduce the >number of file system clients and increase the typical request size. > >On a single processor, though, it's possible the collective overhead >outweighs the benefits. > >How much slower are you seeing? how many MPI processes? what kind of >file view and memory type? > >You might want to look at the src/mpi/romio/test/coll_perf.c test and >experiment with the collective and non-collective versions (you'll have >to modify the test or pass in hints). > >==rob >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Running mpich without mpiexec
by Dmitriy Lyubimov 21 Apr '16

21 Apr '16

Hi, I guess i need to be asking more specific questions. I saw the same issue has been raised a few years ago and apparently there was no solution for it back then, so i thought i'd check again. The question is about bypassing mpiexec and trying to connect already running processes (which already have some data in memory to share with mpi tasks) into a single communicator without having to spawn yet another process with mpiexec. Suppose one of this processes uses MPI_Comm_accept, and the other somehow know the mpi port name and would use MPI_Comm_connect() to connect to it. Everything works as long as both server and client started with mpiexec -n 1. However, without mpiexec, the same attempts cause errors and connection never happens: mpiexec@Intel-Kubu] match_arg (utils/args/args.c:159): unrecognized argument pmi_args [mpiexec@Intel-Kubu] HYDU_parse_array (utils/args/args.c:174): argument matching returned error [mpiexec@Intel-Kubu] parse_args (ui/mpich/utils.c:1596): error parsing input array [mpiexec@Intel-Kubu] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1648): unable to parse user arguments [mpiexec@Intel-Kubu] main (ui/mpich/mpiexec.c:153): error parsing parameters So, there's no way of working around the necessity to spawn an entirely new process with mpiexec? Thank you very much for your help. -Dmitriy _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Is it allowed to attach automatic array for remote access with MPI_Win_attach?
by Jeff Hammond 20 Apr '16

20 Apr '16

On Wed, Apr 20, 2016 at 7:31 AM, Maciej Szpindler <m.szpindler(a)icm.edu.pl> wrote: > Many thanks for the correction and explanation, Jeff. Now it is fixed. > > I am wondering, but this is likely not MPICH specific, if my approach > is correct. I would like to replace send-receive halo exchange module > of the larger application with rma pscw scheme. Basic implementation > performs poorly and I were looking for improvement. Code structure > requires this module to initialize memory windows with every halo > exchange. I have tried to address this with dynamic windows but, as > you have pointed, additional synchronization is then required and > potential benefits of pscw approach are gone. > > Should it be expect that the good implementation of pscw scheme compares > with message passing? > > If you find that PSCW with dynamic windows beats Send-Recv, I will be shocked. I'll spare you the implementation details, but there is absolutely no reason why this should happen in MPICH, or in any other normal implementation. The only way I can see it winning is with a shared memory machine where dynamic windows could leverage XPMEM. This assumes you are not using MPI_Accumulate, by the way, since if you were to use MPI_Accumulate, it would likely be more efficient than the equivalent code in Send-Recv, even with a purely active message implementation of RMA. Now, if you can find a way to use MPI_Win_allocate, you might get some benefit, especially within the node, since in that case RMA will translate directly to load-store in shared memory. A different way to optimize halo exchange is with neighborhood collectives. These may not be optimized in MPICH today, but Torsten has a paper showing how they can be optimized and these are more general than most of the optimizations for RMA. Best, Jeff > Reagrds, > Maciej > > W dniu 19.04.2016 o 17:48, Jeff Hammond pisze: > > When you use dynamic windows, you must use the virtual address of the >> remote memory as the offset, That means you must attach a buffer and then >> get the address with MPI_GET_ADDRESS. Then you must share that address >> with any processes that target that memory, perhaps using >> MPI_SEND/MPI_RECV >> or MPI_ALLGATHER of an address-sized integer (MPI_AINT is the MPI type >> corresponding to the MPI_Aint C type). It appears you are not doing this. >> >> This issue should affect you whether you use automatic arrays or heap >> data... >> >> It does not appear to be a problem here, but if you use automatic arrays >> with RMA, you must guarentee that they remain in scope throughout the >> duration of when they will be accessed remotely. I think you are doing >> this sufficiently with a barrier. However, at the point at which you are >> calling barrier to ensure they stay in scope, you lose all of the benefits >> of fine-grain synchronization from PSCW. You might as well just use >> MPI_Win_fence. >> >> There is a sentence in the MPI spec that says that, strictly speaking, >> using memory not allocated by MPI_Alloc_mem (or MPI_Win_allocate(_shared), >> of course) in RMA is not portable, but I don't know any implementation >> that >> actually behaves this way. MPICH has an active-message implementation of >> RMA, which does not care what storage class is involved, up to performance >> differences (interprocess shared memory is faster in some cases). >> >> This is a fairly complicated topic and it is possible that I have been a >> bit crude in summarizing the MPI standard, so I apologize to any MPI Forum >> experts who can find fault in what I've written :-) >> >> Jeff >> >> On Tue, Apr 19, 2016 at 5:51 AM, Maciej Szpindler <m.szpindler(a)icm.edu.pl >> > >> wrote: >> >> This is simplified version of my routine. It may look odd but I am >>> trying to migrate from send/recv scheme to one sided pscw and that >>> is the reason for buffers etc. As long as dynamic windows are not used, >>> it works fine (I believe). When I switch to dynamic windows it fails >>> with segmentation fault. I would appreciate any comment and suggestion >>> how to improve this. >>> >>> SUBROUTINE swap_simple_rma(field, row_length, rows, levels, halo_size) >>> >>> USE mpi >>> >>> IMPLICIT NONE >>> >>> INTEGER, INTENT(IN) :: row_length >>> INTEGER, INTENT(IN) :: rows >>> INTEGER, INTENT(IN) :: levels >>> INTEGER, INTENT(IN) :: halo_size >>> REAL(KIND=8), INTENT(INOUT) :: field(1:row_length, 1:rows, levels) >>> REAL(KIND=8) :: send_buffer(halo_size, rows, levels) >>> REAL(KIND=8) :: recv_buffer(halo_size, rows, levels) >>> INTEGER :: buffer_size >>> INTEGER :: i,j,k >>> INTEGER(kind=MPI_INTEGER_KIND) :: ierror >>> INTEGER(kind=MPI_INTEGER_KIND) :: my_rank, comm_size >>> Integer(kind=MPI_INTEGER_KIND) :: win, win_info >>> Integer(kind=MPI_INTEGER_KIND) :: my_group, origin_group, target_group >>> Integer(kind=MPI_INTEGER_KIND), DIMENSION(1) :: target_rank, origin_rank >>> Integer(kind=MPI_ADDRESS_KIND) :: win_size, disp >>> >>> CALL MPI_Comm_Rank(MPI_COMM_WORLD, my_rank, ierror) >>> CALL MPI_Comm_Size(MPI_COMM_WORLD, comm_size, ierror) >>> >>> buffer_size = halo_size * rows * levels >>> >>> CALL MPI_Info_create(win_info, ierror) >>> CALL MPI_Info_set(win_info, "no_locks", "true", ierror) >>> >>> CALL MPI_Comm_group(MPI_COMM_WORLD, my_group, ierror) >>> >>> If (my_rank /= comm_size - 1) Then >>> origin_rank = my_rank + 1 >>> CALL MPI_Group_incl(my_group, 1, origin_rank, origin_group, ierror) >>> win_size = 8*buffer_size >>> Else >>> origin_group = MPI_GROUP_EMPTY >>> win_size = 0 >>> End If >>> >>> CALL MPI_Win_create_dynamic(win_info, MPI_COMM_WORLD, win, ierror) >>> !! CALL MPI_Win_create(recv_buffer, win_size, & >>> !! 8, win_info, MPI_COMM_WORLD, win, ierror) >>> CALL MPI_Win_attach(win, recv_buffer, win_size, ierror) >>> >>> CALL MPI_Barrier(MPI_COMM_WORLD, ierror) >>> >>> CALL MPI_Win_post(origin_group, MPI_MODE_NOSTORE, win, ierror) >>> >>> ! Prepare buffer >>> DO k=1,levels >>> DO j=1,rows >>> DO i=1,halo_size >>> send_buffer(i,j,k)=field(i,j,k) >>> END DO ! I >>> END DO ! J >>> END DO ! K >>> >>> If (my_rank /= 0 ) Then >>> target_rank = my_rank - 1 >>> CALL MPI_Group_incl(my_group, 1, target_rank, target_group, ierror) >>> Else >>> target_group = MPI_GROUP_EMPTY >>> End If >>> >>> CALL MPI_Win_start(target_group, 0, win, ierror) >>> >>> disp = 0 >>> >>> If (my_rank /= 0) Then >>> CALL MPI_Put(send_buffer, buffer_size, MPI_REAL8, & >>> my_rank - 1, disp, buffer_size, MPI_REAL8, win, ierror) >>> End If >>> CALL MPI_Win_complete(win, ierror) >>> >>> CALL MPI_Barrier(MPI_COMM_WORLD, ierror) >>> write (0,*) 'Put OK' >>> CALL MPI_Barrier(MPI_COMM_WORLD, ierror) >>> >>> CALL MPI_Win_wait(win, ierror) >>> >>> ! Read from buffer >>> If (my_rank /= comm_size -1 ) Then >>> DO k=1,levels >>> DO j=1,rows >>> DO i=1,halo_size >>> field(row_length+i,j,k) = recv_buffer(i,j,k) >>> END DO >>> END DO >>> END DO >>> End if >>> >>> CALL MPI_Win_detach(win, recv_buffer, ierror) >>> CALL MPI_Win_free(win, ierror) >>> >>> END SUBROUTINE swap_simple_rma >>> >>> Best Regards, >>> Maciej >>> >>> W dniu 14.04.2016 o 19:21, Thakur, Rajeev pisze: >>> >>> After the Win_attach, did you add a barrier or some other form of >>> >>>> synchronization? The put shouldn’t happen before Win_attach returns. >>>> >>>> Rajeev >>>> >>>> On Apr 14, 2016, at 10:56 AM, Maciej Szpindler <m.szpindler(a)icm.edu.pl> >>>> >>>>> wrote: >>>>> >>>>> Dear All, >>>>> >>>>> I am trying to use dynamic RMA windows in fortran. In my case I would >>>>> like to attach automatic array to dynamic window. The question is if >>>>> it is correct and allowed in MPICH. I feel that it is not working, at >>>>> least in cray-mpich/7.3.2. >>>>> >>>>> I have a subroutine that use RMA windows: >>>>> >>>>> SUBROUTINE foo(x, y, z , ...) >>>>> >>>>> USE mpi >>>>> ... >>>>> >>>>> INTEGER, INTENT(IN) :: x, y, z >>>>> REAL(KIND=8) :: buffer(x, y, z) >>>>> INTEGER(kind=MPI_INTEGER_KIND) :: win_info, win, comm >>>>> INTEGER(kind=MPI_INTEGER_KIND) :: buff_size >>>>> ... >>>>> >>>>> buff_size = x*y*z*8 >>>>> >>>>> CALL MPI_Info_create(win_info, ierror) >>>>> CALL MPI_Info_set(win_info, "no_locks", "true", ierror) >>>>> >>>>> CALL MPI_Win_create_dynamic(win_info, comm, win, ierror) >>>>> >>>>> CALL MPI_Win_attach(win, buffer, buff_size, ierror) >>>>> >>>>> ... >>>>> >>>>> This produces segmentation fault when MPI_Put is called on a window, >>>>> while exactly the same routine code with static MPI_Win_create on >>>>> buffer instead of create_dynamic+attach works fine. As far as I >>>>> understand buffer is in this case "simply contiguous" in a sense of >>>>> the MPI Standard. Any help would be appreciated! >>>>> >>>>> Best Regards, >>>>> Maciej >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> _______________________________________________ >>>> >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Thank you for your help
by Dmitriy Lyubimov 20 Apr '16

20 Apr '16

Also. Where can i find reference for PMI_xx standard? Can i launch a worker process defined in an .so which i can load in an already running process i can share memory with? In other words, is there example how i can do the same thing in a .so that the hydra_rmi_proxy does? thank you. -d On Tue, Apr 19, 2016 at 9:21 AM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> wrote: > Hello, > > I am new to MPI. I have a couple of questions I hope someone could help > me with -- i tried to scan archives but have not immediately found the > answers. > > (1) I know there is a multicast based implementation for infiniband in a > derivative MVAPICH peojwxr from Ohio state that apparently allows to avoid > packet duplication for multiple addressees when implementing broadcasts and > scatters, in a reliable communication manner even for larger messages (as > large as perhaps 16...200Mb) > > Is there any similar technique used in MPICH for broadcasts and gathers? > Would there be any networking benefit in using MPICH as opposed to > carefully organized point-to-point broadcast scheme such as perhaps chained > peer-to-peer or butterfly mixing? > > (2) Another question is possibility of integration of the worker processes > with another resource manager (other than Hydra, i guess) -- perhaps yarn > or mesos. Can i bring the worker process first somehow and then enroll it > into a communicator, assuming at certain point i know all process locations > in the network? > > In other words, can i replace whatever mpirun or mpiexec does, with a > custom code that would enroll all workers into the communicators and > assigns the ranks? > > (3) network multinenancy: can i set up several world communicators on the > same network independently at different point of times? or it has to be > only one world communicator at a time? > > Thank you very much for your help. > -Dmitriy > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Is it allowed to attach automatic array for remote access with MPI_Win_attach?
by Maciej Szpindler 20 Apr '16

20 Apr '16

Many thanks for the correction and explanation, Jeff. Now it is fixed. I am wondering, but this is likely not MPICH specific, if my approach is correct. I would like to replace send-receive halo exchange module of the larger application with rma pscw scheme. Basic implementation performs poorly and I were looking for improvement. Code structure requires this module to initialize memory windows with every halo exchange. I have tried to address this with dynamic windows but, as you have pointed, additional synchronization is then required and potential benefits of pscw approach are gone. Should it be expect that the good implementation of pscw scheme compares with message passing? Reagrds, Maciej W dniu 19.04.2016 o 17:48, Jeff Hammond pisze: > When you use dynamic windows, you must use the virtual address of the > remote memory as the offset, That means you must attach a buffer and then > get the address with MPI_GET_ADDRESS. Then you must share that address > with any processes that target that memory, perhaps using MPI_SEND/MPI_RECV > or MPI_ALLGATHER of an address-sized integer (MPI_AINT is the MPI type > corresponding to the MPI_Aint C type). It appears you are not doing this. > > This issue should affect you whether you use automatic arrays or heap > data... > > It does not appear to be a problem here, but if you use automatic arrays > with RMA, you must guarentee that they remain in scope throughout the > duration of when they will be accessed remotely. I think you are doing > this sufficiently with a barrier. However, at the point at which you are > calling barrier to ensure they stay in scope, you lose all of the benefits > of fine-grain synchronization from PSCW. You might as well just use > MPI_Win_fence. > > There is a sentence in the MPI spec that says that, strictly speaking, > using memory not allocated by MPI_Alloc_mem (or MPI_Win_allocate(_shared), > of course) in RMA is not portable, but I don't know any implementation that > actually behaves this way. MPICH has an active-message implementation of > RMA, which does not care what storage class is involved, up to performance > differences (interprocess shared memory is faster in some cases). > > This is a fairly complicated topic and it is possible that I have been a > bit crude in summarizing the MPI standard, so I apologize to any MPI Forum > experts who can find fault in what I've written :-) > > Jeff > > On Tue, Apr 19, 2016 at 5:51 AM, Maciej Szpindler <m.szpindler(a)icm.edu.pl> > wrote: > >> This is simplified version of my routine. It may look odd but I am >> trying to migrate from send/recv scheme to one sided pscw and that >> is the reason for buffers etc. As long as dynamic windows are not used, >> it works fine (I believe). When I switch to dynamic windows it fails >> with segmentation fault. I would appreciate any comment and suggestion >> how to improve this. >> >> SUBROUTINE swap_simple_rma(field, row_length, rows, levels, halo_size) >> >> USE mpi >> >> IMPLICIT NONE >> >> INTEGER, INTENT(IN) :: row_length >> INTEGER, INTENT(IN) :: rows >> INTEGER, INTENT(IN) :: levels >> INTEGER, INTENT(IN) :: halo_size >> REAL(KIND=8), INTENT(INOUT) :: field(1:row_length, 1:rows, levels) >> REAL(KIND=8) :: send_buffer(halo_size, rows, levels) >> REAL(KIND=8) :: recv_buffer(halo_size, rows, levels) >> INTEGER :: buffer_size >> INTEGER :: i,j,k >> INTEGER(kind=MPI_INTEGER_KIND) :: ierror >> INTEGER(kind=MPI_INTEGER_KIND) :: my_rank, comm_size >> Integer(kind=MPI_INTEGER_KIND) :: win, win_info >> Integer(kind=MPI_INTEGER_KIND) :: my_group, origin_group, target_group >> Integer(kind=MPI_INTEGER_KIND), DIMENSION(1) :: target_rank, origin_rank >> Integer(kind=MPI_ADDRESS_KIND) :: win_size, disp >> >> CALL MPI_Comm_Rank(MPI_COMM_WORLD, my_rank, ierror) >> CALL MPI_Comm_Size(MPI_COMM_WORLD, comm_size, ierror) >> >> buffer_size = halo_size * rows * levels >> >> CALL MPI_Info_create(win_info, ierror) >> CALL MPI_Info_set(win_info, "no_locks", "true", ierror) >> >> CALL MPI_Comm_group(MPI_COMM_WORLD, my_group, ierror) >> >> If (my_rank /= comm_size - 1) Then >> origin_rank = my_rank + 1 >> CALL MPI_Group_incl(my_group, 1, origin_rank, origin_group, ierror) >> win_size = 8*buffer_size >> Else >> origin_group = MPI_GROUP_EMPTY >> win_size = 0 >> End If >> >> CALL MPI_Win_create_dynamic(win_info, MPI_COMM_WORLD, win, ierror) >> !! CALL MPI_Win_create(recv_buffer, win_size, & >> !! 8, win_info, MPI_COMM_WORLD, win, ierror) >> CALL MPI_Win_attach(win, recv_buffer, win_size, ierror) >> >> CALL MPI_Barrier(MPI_COMM_WORLD, ierror) >> >> CALL MPI_Win_post(origin_group, MPI_MODE_NOSTORE, win, ierror) >> >> ! Prepare buffer >> DO k=1,levels >> DO j=1,rows >> DO i=1,halo_size >> send_buffer(i,j,k)=field(i,j,k) >> END DO ! I >> END DO ! J >> END DO ! K >> >> If (my_rank /= 0 ) Then >> target_rank = my_rank - 1 >> CALL MPI_Group_incl(my_group, 1, target_rank, target_group, ierror) >> Else >> target_group = MPI_GROUP_EMPTY >> End If >> >> CALL MPI_Win_start(target_group, 0, win, ierror) >> >> disp = 0 >> >> If (my_rank /= 0) Then >> CALL MPI_Put(send_buffer, buffer_size, MPI_REAL8, & >> my_rank - 1, disp, buffer_size, MPI_REAL8, win, ierror) >> End If >> CALL MPI_Win_complete(win, ierror) >> >> CALL MPI_Barrier(MPI_COMM_WORLD, ierror) >> write (0,*) 'Put OK' >> CALL MPI_Barrier(MPI_COMM_WORLD, ierror) >> >> CALL MPI_Win_wait(win, ierror) >> >> ! Read from buffer >> If (my_rank /= comm_size -1 ) Then >> DO k=1,levels >> DO j=1,rows >> DO i=1,halo_size >> field(row_length+i,j,k) = recv_buffer(i,j,k) >> END DO >> END DO >> END DO >> End if >> >> CALL MPI_Win_detach(win, recv_buffer, ierror) >> CALL MPI_Win_free(win, ierror) >> >> END SUBROUTINE swap_simple_rma >> >> Best Regards, >> Maciej >> >> W dniu 14.04.2016 o 19:21, Thakur, Rajeev pisze: >> >> After the Win_attach, did you add a barrier or some other form of >>> synchronization? The put shouldn’t happen before Win_attach returns. >>> >>> Rajeev >>> >>> On Apr 14, 2016, at 10:56 AM, Maciej Szpindler <m.szpindler(a)icm.edu.pl> >>>> wrote: >>>> >>>> Dear All, >>>> >>>> I am trying to use dynamic RMA windows in fortran. In my case I would >>>> like to attach automatic array to dynamic window. The question is if >>>> it is correct and allowed in MPICH. I feel that it is not working, at >>>> least in cray-mpich/7.3.2. >>>> >>>> I have a subroutine that use RMA windows: >>>> >>>> SUBROUTINE foo(x, y, z , ...) >>>> >>>> USE mpi >>>> ... >>>> >>>> INTEGER, INTENT(IN) :: x, y, z >>>> REAL(KIND=8) :: buffer(x, y, z) >>>> INTEGER(kind=MPI_INTEGER_KIND) :: win_info, win, comm >>>> INTEGER(kind=MPI_INTEGER_KIND) :: buff_size >>>> ... >>>> >>>> buff_size = x*y*z*8 >>>> >>>> CALL MPI_Info_create(win_info, ierror) >>>> CALL MPI_Info_set(win_info, "no_locks", "true", ierror) >>>> >>>> CALL MPI_Win_create_dynamic(win_info, comm, win, ierror) >>>> >>>> CALL MPI_Win_attach(win, buffer, buff_size, ierror) >>>> >>>> ... >>>> >>>> This produces segmentation fault when MPI_Put is called on a window, >>>> while exactly the same routine code with static MPI_Win_create on >>>> buffer instead of create_dynamic+attach works fine. As far as I >>>> understand buffer is in this case "simply contiguous" in a sense of >>>> the MPI Standard. Any help would be appreciated! >>>> >>>> Best Regards, >>>> Maciej >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] collective read slower than non-collective read
by Rob Latham 19 Apr '16

19 Apr '16

On 04/19/2016 07:45 AM, 冰 wrote: > Hi, > I run programs in one computer, In my tests here, the MPI_File_read_all > have been slower than MPI_File_read, could you send me a simple routine > that collective read is faster than non-collective read? Need more information. The collective calls provide two important optimizations: reduce the number of file system clients and increase the typical request size. On a single processor, though, it's possible the collective overhead outweighs the benefits. How much slower are you seeing? how many MPI processes? what kind of file view and memory type? You might want to look at the src/mpi/romio/test/coll_perf.c test and experiment with the collective and non-collective versions (you'll have to modify the test or pass in hints). ==rob _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0