April 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] segment fault when using knem
by Dave Goodell 26 Apr '13

26 Apr '13

On Apr 25, 2013, at 8:03 PM CDT, M Xie <xmxmxie(a)gmail.com> wrote: > I am doing some tests with mpich in a dual-processor SMP server. > Now I use mpich-3.0.3, I also use knem for accelerating intra-node communication. > > The channel I used is nemesis:tcp, and nemesis-local-lmt=knem. > But when I use osu_benchmarks to do the bandwidth test, osu_bw will segment > fault at the 2MB, which is the default value of MPICH_NEM_LMT_DMA_THRESHOLD. > It seems when DMA channel in knem is used, the segment fault occurs. > When I set MPICH_NEM_LMT_DMA_THRESHOLD to a smaller value, such as > 131072, osu_bw will segment fault at 131072. > > I also test NAS Parallell Benchmark. I noticed when DMA channel in knem is used, > sometimes the NPB tests will be freezed after running for a while. > > In the attachment, I list some config and core dump files. > > Did anyone met or solved the similar problems. (I saw this over on the knem list but didn't have time until now to respond) Our knem support hasn't been touched in a long time. I think the version we ship in "contrib" is very old (0.5.0), before the full RDMA-style interface existed. Our side of the code probably hasn't been updated since that timeframe, modulo a patch that Brice contributed. Also, we don't run it in our nightly tests, so it's possible it has bit-rotted for another reason. It looks like there's a bad request handle being dereferenced, though just from the stack trace it's hard to tell whether that's a corrupted handle (garbage) or a stale handle value which has been freed. We could be hitting this longstanding LMT bug: https://trac.mpich.org/projects/mpich/ticket/1039 Let me try to setup an environment where I can reproduce this and see if there's an easy/obvious fix. -Dave

1 0

segment fault when using knem
by M Xie 25 Apr '13

25 Apr '13

Hello, I am doing some tests with mpich in a dual-processor SMP server. Now I use mpich-3.0.3, I also use knem for accelerating intra-node communication. The channel I used is nemesis:tcp, and nemesis-local-lmt=knem. But when I use osu_benchmarks to do the bandwidth test, osu_bw will segment fault at the 2MB, which is the default value of MPICH_NEM_LMT_DMA_THRESHOLD. It seems when DMA channel in knem is used, the segment fault occurs. When I set MPICH_NEM_LMT_DMA_THRESHOLD to a smaller value, such as 131072, osu_bw will segment fault at 131072. I also test NAS Parallell Benchmark. I noticed when DMA channel in knem is used, sometimes the NPB tests will be freezed after running for a while. In the attachment, I list some config and core dump files. Did anyone met or solved the similar problems. Thanks for your help.

1 0

Re: [mpich-discuss] MPICH 3.0.4 Ubuntu, Debian and Slackware packages (x86 and amd64)
by Pavan Balaji 25 Apr '13

25 Apr '13

1 0

MPICH 3.0.4 Ubuntu, Debian and Slackware packages (x86 and amd64)
by Marcelo Soares Souza 25 Apr '13

25 Apr '13

1 0

Retagged mpich-3.0.4
by Pavan Balaji 25 Apr '13

25 Apr '13

All, We accidentally tagged mpich-3.0.4 incorrectly in the mpich git repo to point to [85acf6c5], instead of the correct hash [0f1067da]. The release tarball is still fine; just the git tag was incorrect. We have fixed this in the git repo now. If you have pulled the 3.0.4 tag from the mpich git repo, please correct your cloned copy using: % git tag -d v3.0.4 % git fetch origin tag v3.0.4 You might also be able to do: % git pull --tags (though we are not entirely sure that'll work). Sorry about the mess-up. Once again, the release tarball is correct. Just the tag was incorrect. Regards, The MPICH Team -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Query about MPI_Bsend
by Vihari Piratla 25 Apr '13

25 Apr '13

I am trying to implement a logical ring with MPI; where every process receives a message from process with id one less than that of current process and forwards to the next process in cyclic fashion. My aim is to traffic buffer such that it loses some messages or maybe puts them in out of order. A cycle of communication will finish when a message dispatched by root node comes back again to root. here is the code that I have tried: I am just including relevant parts of it. if(procId!=root) { sleep(100); while(1) { tm = MPI_Wtime(); MPI_Irecv( &message, STR_LEN, MPI_CHAR, ((procId-1)>=0?(procId-1):(numProc-1)),RETURN_DATA_TAG, MPI_COMM_WORLD,&receiveRequest); MPI_Wait(&receiveRequest,&status); printf("%d: Received\n",procId); if(!strncmp(message,"STOP",4)&&(procId==(numProc-1))) break; MPI_Ssend( message, STR_LEN, MPI_CHAR, (procId+1)%numProc, SEND_DATA_TAG, MPI_COMM_WORLD); if(!strncmp(message,"STOP",4)) break; printf("%d: Sent\n",procId); } } else { for(iter=0;iter<benchmarkSize;iter++) { //Synthesize the message message[STR_LEN-1] = '\0'; iErr = MPI_Bsend( message, STR_LEN, MPI_CHAR, (root+1)%numProc, SEND_DATA_TAG, MPI_COMM_WORLD); if (iErr != MPI_SUCCESS) { char error_string[BUFSIZ]; int length_of_error_string; MPI_Error_string(iErr, error_string, &length_of_error_string); fprintf(stderr, "%3d: %s\n", procId, error_string); } tm = MPI_Wtime(); while(((MPI_Wtime()-tm)*1000)<delay); printf("Root: Sending\n"); } for(iter=0;iter<benchmarkSize;iter++) { MPI_Recv(message,STR_LEN,MPI_CHAR, (numProc-1),RETURN_DATA_TAG,MPI_COMM_WORLD,&status); //We should not wait for the messages to be received but wait for certain amount of time //Extract the fields in the message if(((prevRcvdSeqNum+1)!=atoi(seqNum))&&(prevRcvdSeqNum!=0)) outOfOrderMsgs++; prevRcvdSeqNum = atoi(seqNum); printf("Seq Num: %d\n",atoi(seqNum)); rcvdMsgs++; printf("Root: Receiving\n"); } MPI_Isend( "STOP", 4, MPI_CHAR, (root+1)%numProc, SEND_DATA_TAG, MPI_COMM_WORLD,&sendRequest); MPI_Wait(&sendRequest,&status); /*This is to ask all other processes to terminate, when the work is done*/ } Now, I have these questions: 1) Why is it that when I inject some sleep in the other processes(I mean other than root) of the ring; NO receive is taking place? 2) Even when the buffer size is only one, how is it that root node is able to dispatch messages through MPI_Bsend without an error? for example the case when it needs to send total 10 messages at a rate of 1000 per second and with buffer size of 1. MPI_Bsend is able to dispatch all the messages without any error of "buffer full"; irrespective of the presence of sleep() in other processes of the ring! In short How on earth is MPI_Bsend is not giving me an error "Insufficient space in buffer" even if my buffersize is size enough to accommodate only a single message, MPI_Bsend did not show an error for 65 of the total 100 messages sent? I intentionally made the receive buffers to be unavailable by making other processes to sleep before they start processing receive and send. Thanks a ton! -- V

1 0

Re: [mpich-discuss] non-blocking communication and subarray data type
by Dave Goodell 22 Apr '13

22 Apr '13

Configure your MPICH with "--enable-g" and you should get some error checks to help with this. If you point it at your valgrind headers then you might get even more useful output from Valgrind. Something like "--with-valgrind=/Users/goodell/prefix/include" at configure time. Based on the stack trace you posted, MPI certainly thinks that the refcount has dropped to 0. -Dave On Apr 22, 2013, at 6:06 PM CDT, burlen <burlen.loring(a)gmail.com> wrote: > Hi Jim, > > That's what my vague recollection was, however valgrind is reporting that the object was free'd when I called type free, and MPI later crashes when starts using that data. In the report bellow vtkPixelTransfer::Execute is where I call irecv and then type free. > > I'm wondering if there a way to access the reference count? I'd like to verify that it's not 0 after I call type free. > > Burlen > > ==9094== Invalid read of size 4 > ==9094== at 0x1A0FF16C: MPID_Segment_manipulate (segment.c:429) > ==9094== by 0x1A108280: MPID_Segment_pack (segment_packunpack.c:63) > ==9094== by 0x1A0D1A8B: MPIDI_CH3I_Shm_send_progress (mpid_nem_inline.h:458) > ==9094== by 0x1A0D42F1: MPIDI_CH3I_Progress (ch3_progress.c:438) > ==9094== by 0x1A1DE5BB: PMPI_Waitany (waitany.c:198) > ==9094== by 0x103D0692: vtkSurfaceLICComposite::Gather(void*, int, int, vtkTextureObject*&) (vtkSurfaceLICComposite.cxx:1511) > ==9094== by 0x103F0673: vtkSurfaceLICPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkSurfaceLICPainter.cxx:2944) > ==9094== by 0x156D9A1E: vtkPainter::Render(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:218) > ==9094== by 0x156D9A95: vtkPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:228) > ==9094== by 0x125D0D09: vtkOpenGLClipPlanesPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkOpenGLClipPlanesPainter.cxx:101) > ==9094== by 0x156D9A1E: vtkPainter::Render(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:218) > ==9094== by 0x156D9A95: vtkPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:228) > ==9094== Address 0x284dcc18 is 152 bytes inside a block of size 384 free'd > ==9094== at 0x4C2B7BE: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) > ==9094== by 0x20E8D1C7: MPL_trfree (in /work/apps/mpich-3.0.3-D/lib/libmpl.so.1.0.0) > ==9094== by 0x1A0689DC: MPIU_trfree (trmem.c:37) > ==9094== by 0x1A0F984E: MPID_Dataloop_free (dataloop.c:71) > ==9094== by 0x1A109E9A: MPID_Datatype_free (mpid_datatype_free.c:51) > ==9094== by 0x1A19B577: MPIR_Type_free_impl (type_free.c:35) > ==9094== by 0x1A19B9BC: PMPI_Type_free (type_free.c:138) > ==9094== by 0x10389EB4: int vtkPixelTransfer::Execute<float, float>(int, int, int, float*, float*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.h:607) > ==9094== by 0x10382D3F: int vtkPixelTransfer::Execute<float>(int, int, int, float*, int, void*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.h:454) > ==9094== by 0x10382135: vtkPixelTransfer::Execute(int, int, int, int, void*, int, void*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.cxx:33) > ==9094== by 0x103CFD08: vtkSurfaceLICComposite::Gather(void*, int, int, vtkTextureObject*&) (vtkSurfaceLICComposite.cxx:1396) > ==9094== by 0x103F0673: vtkSurfaceLICPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkSurfaceLICPainter.cxx:2944) > > > > On 04/22/2013 02:40 PM, Jim Dinan wrote: >> Hi Burlen, >> >> You can free the datatype as soon as you are finished using it. MPI will internally maintain a reference to the datatype so that it is not actually freed until both the MPI library and the application code are finished using it. >> >> ~Jim. >> >> On 4/22/13 4:29 PM, burlen wrote: >>> Can anyone clarify about using subarray datatype with irecv, When is ok >>> to free the subarray data type? Can I do so immediately after calling >>> irecv? or must I wait until the recv really finishes? I didn't see any >>> specific language about this in the documentation , but I'm experiencing >>> segfaults when MPI accesses previously free'd memory in the datatype. >>> >>> Thanks >>> Burlen >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] non-blocking communication and subarray data type
by burlen 22 Apr '13

22 Apr '13

Hi Jim, That's what my vague recollection was, however valgrind is reporting that the object was free'd when I called type free, and MPI later crashes when starts using that data. In the report bellow vtkPixelTransfer::Execute is where I call irecv and then type free. I'm wondering if there a way to access the reference count? I'd like to verify that it's not 0 after I call type free. Burlen ==9094== Invalid read of size 4 ==9094== at 0x1A0FF16C: MPID_Segment_manipulate (segment.c:429) ==9094== by 0x1A108280: MPID_Segment_pack (segment_packunpack.c:63) ==9094== by 0x1A0D1A8B: MPIDI_CH3I_Shm_send_progress (mpid_nem_inline.h:458) ==9094== by 0x1A0D42F1: MPIDI_CH3I_Progress (ch3_progress.c:438) ==9094== by 0x1A1DE5BB: PMPI_Waitany (waitany.c:198) ==9094== by 0x103D0692: vtkSurfaceLICComposite::Gather(void*, int, int, vtkTextureObject*&) (vtkSurfaceLICComposite.cxx:1511) ==9094== by 0x103F0673: vtkSurfaceLICPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkSurfaceLICPainter.cxx:2944) ==9094== by 0x156D9A1E: vtkPainter::Render(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:218) ==9094== by 0x156D9A95: vtkPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:228) ==9094== by 0x125D0D09: vtkOpenGLClipPlanesPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkOpenGLClipPlanesPainter.cxx:101) ==9094== by 0x156D9A1E: vtkPainter::Render(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:218) ==9094== by 0x156D9A95: vtkPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkPainter.cxx:228) ==9094== Address 0x284dcc18 is 152 bytes inside a block of size 384 free'd ==9094== at 0x4C2B7BE: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==9094== by 0x20E8D1C7: MPL_trfree (in /work/apps/mpich-3.0.3-D/lib/libmpl.so.1.0.0) ==9094== by 0x1A0689DC: MPIU_trfree (trmem.c:37) ==9094== by 0x1A0F984E: MPID_Dataloop_free (dataloop.c:71) ==9094== by 0x1A109E9A: MPID_Datatype_free (mpid_datatype_free.c:51) ==9094== by 0x1A19B577: MPIR_Type_free_impl (type_free.c:35) ==9094== by 0x1A19B9BC: PMPI_Type_free (type_free.c:138) ==9094== by 0x10389EB4: int vtkPixelTransfer::Execute<float, float>(int, int, int, float*, float*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.h:607) ==9094== by 0x10382D3F: int vtkPixelTransfer::Execute<float>(int, int, int, float*, int, void*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.h:454) ==9094== by 0x10382135: vtkPixelTransfer::Execute(int, int, int, int, void*, int, void*, std::vector<int, std::allocator<int> >&, int) (vtkPixelTransfer.cxx:33) ==9094== by 0x103CFD08: vtkSurfaceLICComposite::Gather(void*, int, int, vtkTextureObject*&) (vtkSurfaceLICComposite.cxx:1396) ==9094== by 0x103F0673: vtkSurfaceLICPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) (vtkSurfaceLICPainter.cxx:2944) On 04/22/2013 02:40 PM, Jim Dinan wrote: > Hi Burlen, > > You can free the datatype as soon as you are finished using it. MPI > will internally maintain a reference to the datatype so that it is not > actually freed until both the MPI library and the application code are > finished using it. > > ~Jim. > > On 4/22/13 4:29 PM, burlen wrote: >> Can anyone clarify about using subarray datatype with irecv, When is ok >> to free the subarray data type? Can I do so immediately after calling >> irecv? or must I wait until the recv really finishes? I didn't see any >> specific language about this in the documentation , but I'm experiencing >> segfaults when MPI accesses previously free'd memory in the datatype. >> >> Thanks >> Burlen >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 library inconsistencies
by Dave Goodell 22 Apr '13

22 Apr '13

On Apr 22, 2013, at 12:19 PM CDT, Markus Geimer <m.geimer(a)fz-juelich.de> wrote: >> It looks like weak symbol detection on clang is broken/missing. If >> you can, please send the config.log from your configure step (and >> ideally the c.txt file mentioned in the README as well). > > Please find the requested files (for MPICH 3.0.3) attached. The few > warnings in 'm.txt' might be of interest as well (I compiled MPICH > inside a 32-bit chroot environment we are using to find potential > portability issues in our software). > > If you need something else, or have a patch for me to try out, please > let me know. Tracking this here: https://trac.mpich.org/projects/mpich/ticket/1815 Thanks for letting us know, -Dave

1 0

Re: [mpich-discuss] non-blocking communication and subarray data type
by Jim Dinan 22 Apr '13

22 Apr '13

Hi Burlen, You can free the datatype as soon as you are finished using it. MPI will internally maintain a reference to the datatype so that it is not actually freed until both the MPI library and the application code are finished using it. ~Jim. On 4/22/13 4:29 PM, burlen wrote: > Can anyone clarify about using subarray datatype with irecv, When is ok > to free the subarray data type? Can I do so immediately after calling > irecv? or must I wait until the recv really finishes? I didn't see any > specific language about this in the documentation , but I'm experiencing > segfaults when MPI accesses previously free'd memory in the datatype. > > Thanks > Burlen > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0