- discuss - mailman3.cels.anl.gov

Query about MPI_Bsend
by Vihari Piratla 25 Apr '13

25 Apr '13

I am trying to implement a logical ring with MPI; where every process receives a message from process with id one less than that of current process and forwards to the next process in cyclic fashion. My aim is to traffic buffer such that it loses some messages or maybe puts them in out of order. A cycle of communication will finish when a message dispatched by root node comes back again to root. here is the code that I have tried: I am just including relevant parts of it. if(procId!=root) { sleep(100); while(1) { tm = MPI_Wtime(); MPI_Irecv( &message, STR_LEN, MPI_CHAR, ((procId-1)>=0?(procId-1):(numProc-1)),RETURN_DATA_TAG, MPI_COMM_WORLD,&receiveRequest); MPI_Wait(&receiveRequest,&status); printf("%d: Received\n",procId); if(!strncmp(message,"STOP",4)&&(procId==(numProc-1))) break; MPI_Ssend( message, STR_LEN, MPI_CHAR, (procId+1)%numProc, SEND_DATA_TAG, MPI_COMM_WORLD); if(!strncmp(message,"STOP",4)) break; printf("%d: Sent\n",procId); } } else { for(iter=0;iter<benchmarkSize;iter++) { //Synthesize the message message[STR_LEN-1] = '\0'; iErr = MPI_Bsend( message, STR_LEN, MPI_CHAR, (root+1)%numProc, SEND_DATA_TAG, MPI_COMM_WORLD); if (iErr != MPI_SUCCESS) { char error_string[BUFSIZ]; int length_of_error_string; MPI_Error_string(iErr, error_string, &length_of_error_string); fprintf(stderr, "%3d: %s\n", procId, error_string); } tm = MPI_Wtime(); while(((MPI_Wtime()-tm)*1000)<delay); printf("Root: Sending\n"); } for(iter=0;iter<benchmarkSize;iter++) { MPI_Recv(message,STR_LEN,MPI_CHAR, (numProc-1),RETURN_DATA_TAG,MPI_COMM_WORLD,&status); //We should not wait for the messages to be received but wait for certain amount of time //Extract the fields in the message if(((prevRcvdSeqNum+1)!=atoi(seqNum))&&(prevRcvdSeqNum!=0)) outOfOrderMsgs++; prevRcvdSeqNum = atoi(seqNum); printf("Seq Num: %d\n",atoi(seqNum)); rcvdMsgs++; printf("Root: Receiving\n"); } MPI_Isend( "STOP", 4, MPI_CHAR, (root+1)%numProc, SEND_DATA_TAG, MPI_COMM_WORLD,&sendRequest); MPI_Wait(&sendRequest,&status); /*This is to ask all other processes to terminate, when the work is done*/ } Now, I have these questions: 1) Why is it that when I inject some sleep in the other processes(I mean other than root) of the ring; NO receive is taking place? 2) Even when the buffer size is only one, how is it that root node is able to dispatch messages through MPI_Bsend without an error? for example the case when it needs to send total 10 messages at a rate of 1000 per second and with buffer size of 1. MPI_Bsend is able to dispatch all the messages without any error of "buffer full"; irrespective of the presence of sleep() in other processes of the ring! In short How on earth is MPI_Bsend is not giving me an error "Insufficient space in buffer" even if my buffersize is size enough to accommodate only a single message, MPI_Bsend did not show an error for 65 of the total 100 messages sent? I intentionally made the receive buffers to be unavailable by making other processes to sleep before they start processing receive and send. Thanks a ton! -- V

1 0

Re: [mpich-discuss] non-blocking communication and subarray data type
by Dave Goodell 22 Apr '13

22 Apr '13

Configure your MPICH with "--enable-g" and you should get some error checks to help with this. If you point it at your valgrind headers then you might get even more useful output from Valgrind. Something like "--with-valgrind=/Users/goodell/prefix/include" at configure time. Based on the stack trace you posted, MPI certainly thinks that the refcount has dropped to 0. -Dave On Apr 22, 2013, at 6:06 PM CDT, burlen <burlen.loring(a)gmail.com> wrote: > Hi Jim, > > That's what my vague recollection was, however valgrind is reporting that the object was free'd when I called type free, and MPI later crashes when starts using that data. In the report bellow vtkPixelTransfer::Execute is where I call irecv and then type free. > > I'm wondering if there a way to access the reference count? I'd like to verify that it's not 0 after I call type free. > > Burlen > > ==9094== Invalid read of size 4 > ==9094== at 0x1A0FF16C: MPID_Segment_manipulate (segment.c:429) > ==9094== by 0x1A108280: MPID_Segment_pack (segment_packunpack.c:63) > ==9094== by 0x1A0D1A8B: MPIDI_CH3I_Shm_send_progress (mpid_nem_inline.h:458) > ==9094== by 0x1A0D42F1: MPIDI_CH3I_Progress (ch3_progress.c:438) > ==9094== by 0x1A1DE5BB: PMPI_Waitany (waitany.c:198) > ==9094== by 0x103D0692: vtkSurfaceLICComposite::Gather(void*, int, int, vtkTextureObject*&) (vtkSurfaceLICComposite.cxx:1511) > ==9094== by 0x103F0673: vtkSurfaceLICPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkSurfaceLICPainter.cxx:2944) > ==9094== by 0x156D9A1E: vtkPainter::Render(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:218) > ==9094== by 0x156D9A95: vtkPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:228) > ==9094== by 0x125D0D09: vtkOpenGLClipPlanesPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkOpenGLClipPlanesPainter.cxx:101) > ==9094== by 0x156D9A1E: vtkPainter::Render(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:218) > ==9094== by 0x156D9A95: vtkPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:228) > ==9094== Address 0x284dcc18 is 152 bytes inside a block of size 384 free'd > ==9094== at 0x4C2B7BE: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) > ==9094== by 0x20E8D1C7: MPL_trfree (in /work/apps/mpich-3.0.3-D/lib/libmpl.so.1.0.0) > ==9094== by 0x1A0689DC: MPIU_trfree (trmem.c:37) > ==9094== by 0x1A0F984E: MPID_Dataloop_free (dataloop.c:71) > ==9094== by 0x1A109E9A: MPID_Datatype_free (mpid_datatype_free.c:51) > ==9094== by 0x1A19B577: MPIR_Type_free_impl (type_free.c:35) > ==9094== by 0x1A19B9BC: PMPI_Type_free (type_free.c:138) > ==9094== by 0x10389EB4: int vtkPixelTransfer::Execute<float, float>(int, int, int, float*, float*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.h:607) > ==9094== by 0x10382D3F: int vtkPixelTransfer::Execute<float>(int, int, int, float*, int, void*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.h:454) > ==9094== by 0x10382135: vtkPixelTransfer::Execute(int, int, int, int, void*, int, void*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.cxx:33) > ==9094== by 0x103CFD08: vtkSurfaceLICComposite::Gather(void*, int, int, vtkTextureObject*&) (vtkSurfaceLICComposite.cxx:1396) > ==9094== by 0x103F0673: vtkSurfaceLICPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkSurfaceLICPainter.cxx:2944) > > > > On 04/22/2013 02:40 PM, Jim Dinan wrote: >> Hi Burlen, >> >> You can free the datatype as soon as you are finished using it. MPI will internally maintain a reference to the datatype so that it is not actually freed until both the MPI library and the application code are finished using it. >> >> ~Jim. >> >> On 4/22/13 4:29 PM, burlen wrote: >>> Can anyone clarify about using subarray datatype with irecv, When is ok >>> to free the subarray data type? Can I do so immediately after calling >>> irecv? or must I wait until the recv really finishes? I didn't see any >>> specific language about this in the documentation , but I'm experiencing >>> segfaults when MPI accesses previously free'd memory in the datatype. >>> >>> Thanks >>> Burlen >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] non-blocking communication and subarray data type
by burlen 22 Apr '13

22 Apr '13

Hi Jim, That's what my vague recollection was, however valgrind is reporting that the object was free'd when I called type free, and MPI later crashes when starts using that data. In the report bellow vtkPixelTransfer::Execute is where I call irecv and then type free. I'm wondering if there a way to access the reference count? I'd like to verify that it's not 0 after I call type free. Burlen ==9094== Invalid read of size 4 ==9094== at 0x1A0FF16C: MPID_Segment_manipulate (segment.c:429) ==9094== by 0x1A108280: MPID_Segment_pack (segment_packunpack.c:63) ==9094== by 0x1A0D1A8B: MPIDI_CH3I_Shm_send_progress (mpid_nem_inline.h:458) ==9094== by 0x1A0D42F1: MPIDI_CH3I_Progress (ch3_progress.c:438) ==9094== by 0x1A1DE5BB: PMPI_Waitany (waitany.c:198) ==9094== by 0x103D0692: vtkSurfaceLICComposite::Gather(void*, int, int, vtkTextureObject*&) (vtkSurfaceLICComposite.cxx:1511) ==9094== by 0x103F0673: vtkSurfaceLICPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkSurfaceLICPainter.cxx:2944) ==9094== by 0x156D9A1E: vtkPainter::Render(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:218) ==9094== by 0x156D9A95: vtkPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:228) ==9094== by 0x125D0D09: vtkOpenGLClipPlanesPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkOpenGLClipPlanesPainter.cxx:101) ==9094== by 0x156D9A1E: vtkPainter::Render(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:218) ==9094== by 0x156D9A95: vtkPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:228) ==9094== Address 0x284dcc18 is 152 bytes inside a block of size 384 free'd ==9094== at 0x4C2B7BE: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==9094== by 0x20E8D1C7: MPL_trfree (in /work/apps/mpich-3.0.3-D/lib/libmpl.so.1.0.0) ==9094== by 0x1A0689DC: MPIU_trfree (trmem.c:37) ==9094== by 0x1A0F984E: MPID_Dataloop_free (dataloop.c:71) ==9094== by 0x1A109E9A: MPID_Datatype_free (mpid_datatype_free.c:51) ==9094== by 0x1A19B577: MPIR_Type_free_impl (type_free.c:35) ==9094== by 0x1A19B9BC: PMPI_Type_free (type_free.c:138) ==9094== by 0x10389EB4: int vtkPixelTransfer::Execute<float, float>(int, int, int, float*, float*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.h:607) ==9094== by 0x10382D3F: int vtkPixelTransfer::Execute<float>(int, int, int, float*, int, void*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.h:454) ==9094== by 0x10382135: vtkPixelTransfer::Execute(int, int, int, int, void*, int, void*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.cxx:33) ==9094== by 0x103CFD08: vtkSurfaceLICComposite::Gather(void*, int, int, vtkTextureObject*&) (vtkSurfaceLICComposite.cxx:1396) ==9094== by 0x103F0673: vtkSurfaceLICPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkSurfaceLICPainter.cxx:2944) On 04/22/2013 02:40 PM, Jim Dinan wrote: > Hi Burlen, > > You can free the datatype as soon as you are finished using it. MPI > will internally maintain a reference to the datatype so that it is not > actually freed until both the MPI library and the application code are > finished using it. > > ~Jim. > > On 4/22/13 4:29 PM, burlen wrote: >> Can anyone clarify about using subarray datatype with irecv, When is ok >> to free the subarray data type? Can I do so immediately after calling >> irecv? or must I wait until the recv really finishes? I didn't see any >> specific language about this in the documentation , but I'm experiencing >> segfaults when MPI accesses previously free'd memory in the datatype. >> >> Thanks >> Burlen >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 library inconsistencies
by Dave Goodell 22 Apr '13

22 Apr '13

On Apr 22, 2013, at 12:19 PM CDT, Markus Geimer <m.geimer(a)fz-juelich.de> wrote: >> It looks like weak symbol detection on clang is broken/missing. If >> you can, please send the config.log from your configure step (and >> ideally the c.txt file mentioned in the README as well). > > Please find the requested files (for MPICH 3.0.3) attached. The few > warnings in 'm.txt' might be of interest as well (I compiled MPICH > inside a 32-bit chroot environment we are using to find potential > portability issues in our software). > > If you need something else, or have a patch for me to try out, please > let me know. Tracking this here: https://trac.mpich.org/projects/mpich/ticket/1815 Thanks for letting us know, -Dave

1 0

Re: [mpich-discuss] non-blocking communication and subarray data type
by Jim Dinan 22 Apr '13

22 Apr '13

Hi Burlen, You can free the datatype as soon as you are finished using it. MPI will internally maintain a reference to the datatype so that it is not actually freed until both the MPI library and the application code are finished using it. ~Jim. On 4/22/13 4:29 PM, burlen wrote: > Can anyone clarify about using subarray datatype with irecv, When is ok > to free the subarray data type? Can I do so immediately after calling > irecv? or must I wait until the recv really finishes? I didn't see any > specific language about this in the documentation , but I'm experiencing > segfaults when MPI accesses previously free'd memory in the datatype. > > Thanks > Burlen > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

non-blocking communication and subarray data type
by burlen 22 Apr '13

22 Apr '13

Can anyone clarify about using subarray datatype with irecv, When is ok to free the subarray data type? Can I do so immediately after calling irecv? or must I wait until the recv really finishes? I didn't see any specific language about this in the documentation , but I'm experiencing segfaults when MPI accesses previously free'd memory in the datatype. Thanks Burlen

1 0

Re: [mpich-discuss] MPICH3 library inconsistencies
by Markus Geimer 22 Apr '13

22 Apr '13

Hi Dave, > It looks like weak symbol detection on clang is broken/missing. If > you can, please send the config.log from your configure step (and > ideally the c.txt file mentioned in the README as well). Please find the requested files (for MPICH 3.0.3) attached. The few warnings in 'm.txt' might be of interest as well (I compiled MPICH inside a 32-bit chroot environment we are using to find potential portability issues in our software). If you need something else, or have a patch for me to try out, please let me know. Thanks, Markus -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax: +49-2461-61-6656 E-Mail: m.geimer(a)fz-juelich.de WWW: http://www.fz-juelich.de/ias/jsc ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

Re: [mpich-discuss] MPICH3 library inconsistencies
by Dmitri Gribenko 22 Apr '13

22 Apr '13

On Mon, Apr 22, 2013 at 3:44 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > It looks like weak symbol detection on clang is broken/missing. If you can, please send the config.log from your configure step (and ideally the c.txt file mentioned in the README as well). I have found the same bug a few days ago: http://llvm.org/bugs/show_bug.cgi?id=15695 Clang does support other (more common) forms of pragma weak. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr(a)gmail.com>*/

1 0

Re: [mpich-discuss] MPICH3 library inconsistencies
by Dave Goodell 22 Apr '13

22 Apr '13

On Apr 22, 2013, at 2:41 AM CDT, Markus Geimer <m.geimer(a)fz-juelich.de> wrote: > While experimenting with MPICH 3.0.2 and various compilers, > I found an inconsistency which probably isn't intended. > > When configuring MPICH an a Linux system for GNU or Intel > compilers with [snip] > However, when configuring for Clang 3.2 (with GNU Fortran) using > > ./configure --prefix=... --enable-debuginfo \ > CC=clang CXX=clang++ F77=gfortran F90=gfortran [snip] > Obviously, there is an additional library 'libpmpich.a'. > Further investigation reveals that the distribution of > symbols across these libraries is somewhat weird, too: [snip] > My assumption is that something's going wrong during configure > which subsequently messes up the build. Maybe something to > investigate for the next bugfix release... It looks like weak symbol detection on clang is broken/missing. If you can, please send the config.log from your configure step (and ideally the c.txt file mentioned in the README as well). Thanks for letting us know. Many of us use OS X instead of Linux for daily development, which simply does not support weak symbols (in the usual sense), so this is harder for us to notice by accident. -Dave

1 0

MPICH3 library inconsistencies
by Markus Geimer 22 Apr '13

22 Apr '13

Dear MPICH developers, While experimenting with MPICH 3.0.2 and various compilers, I found an inconsistency which probably isn't intended. When configuring MPICH an a Linux system for GNU or Intel compilers with ./configure --prefix=... --enable-debuginfo \ CC=gcc CXX=g++ F77=gfortran F90=gfortran or ./configure --prefix=... --enable-debuginfo \ CC=icc CXX=icpc F77=ifort F90=ifort respectively, the following libraries are built (ignoring the shared libs here for simplicity): ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- $ cd <prefix_{gnu,intel}>/lib $ ls *.a libfmpich.a libmpichcxx.a libmpl.a libtvmpich.a libmpich.a libmpichf90.a libopa.a ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- However, when configuring for Clang 3.2 (with GNU Fortran) using ./configure --prefix=... --enable-debuginfo \ CC=clang CXX=clang++ F77=gfortran F90=gfortran results in ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- $ cd <prefix_clang>/lib $ ls *.a libfmpich.a libmpichcxx.a libmpl.a libpmpich.a libmpich.a libmpichf90.a libopa.a libtvmpich.a ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- Obviously, there is an additional library 'libpmpich.a'. Further investigation reveals that the distribution of symbols across these libraries is somewhat weird, too: ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- $ cd <prefix_clang>/lib $ nm libmpich.a | grep 'T PMPI_Recv' 00000000 T PMPI_Recv 00000000 T PMPI_Recv_init $ nm libpmpich.a | grep 'T PMPI_Recv' $ nm libmpich.a | grep 'T MPI_Recv' $ nm libpmpich.a | grep 'T MPI_Recv' 00000000 T MPI_Recv 00000000 T MPI_Recv_init $ $ nm libmpich.a | grep 'T MPI_File_open' 00000000 T MPI_File_open $ nm libpmpich.a | grep 'T MPI_File_open' 00000000 T MPI_File_open $ nm libmpich.a | grep 'T PMPI_File_open' 00000000 T PMPI_File_open $ nm libpmpich.a | grep 'T PMPI_File_open' 00000000 T PMPI_File_open ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- My assumption is that something's going wrong during configure which subsequently messes up the build. Maybe something to investigate for the next bugfix release... Thanks, Markus -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax: +49-2461-61-6656 E-mail: m.geimer(a)fz-juelich.de WWW: http://www.fz-juelich.de/jsc/ ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0