March 2014 - discuss - mailman3.cels.anl.gov

Is KNEM used for 1-sided comm?
by Velickovic Nikola 25 Mar '14

25 Mar '14

Hello, I am using MPICH 3.0.4 with asynchronous progress enabled along with KNEM module on my system. I have a DMA unit available and it is properly used by KNEM. When I try a benchmark using non-blocking comm. there is computation-communication overlap, but when I try the same using 1-sided communication there is no overlap. It seems like MPICH is not using KNEM for 1-sided comm. Do you have any idea why this might be? Best, Nikola

1 0

Re: [mpich-discuss] ROMIO filesystem check during MPI_File_open
by Jeff Squyres (jsquyres) 25 Mar '14

25 Mar '14

On Mar 25, 2014, at 10:19 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > thanks for the testing. the commit did not mess up jenkins any worse than it's already messed up, so i pushed it to master: > > http://git.mpich.org/mpich.git/commit/dbad7873926a75adbff0fd0140ae321412f70… Thank you! -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpich-discuss] ROMIO filesystem check during MPI_File_open
by Rob Latham 25 Mar '14

25 Mar '14

On 03/24/2014 05:06 PM, Jeff Squyres (jsquyres) wrote: > That seems to do it -- thanks! > > Can you send me the git hash when you commit? When we pull individual patches like this, we like to cite the upstream source where we got it. > thanks for the testing. the commit did not mess up jenkins any worse than it's already messed up, so i pushed it to master: http://git.mpich.org/mpich.git/commit/dbad7873926a75adbff0fd0140ae321412f70… ==rob

1 0

Re: [mpich-discuss] ROMIO filesystem check during MPI_File_open
by Jeff Squyres (jsquyres) 24 Mar '14

24 Mar '14

That seems to do it -- thanks! Can you send me the git hash when you commit? When we pull individual patches like this, we like to cite the upstream source where we got it. Many thanks. On Mar 24, 2014, at 5:54 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 03/24/2014 04:37 PM, Jeff Squyres (jsquyres) wrote: >> On Mar 24, 2014, at 5:17 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: >> >>> Every process will call ADIO_FileSysType_fncall() in the case where have_nfs_enabled==1 . Your interpretation of the code is so different from mine that I'm just going to have to past what I'm looking at and you can tell me where I'm wrong. These lines come from MPICH's ad_fstype.c. The content is the same in openmpi-1.6.4, just shifted by 25 or so lines: >>> >>> 637 ADIO_FileSysType_fncall(filename, &file_system, &myerrcode); >>> 638 if (myerrcode != MPI_SUCCESS) { >> >> This is where our disconnect is occurring: _fncall returns myerrcode==MPI_SUCCESS in all cases. >> >> Meaning: the block with the Allreduce is not entered. This causes everyone to just use the file_system value they got from _fncall: one will have ADIO_UFS, and the rest will have ADIO_NFS. >> >> /Me checks again just to make sure I'm not goofing this up... Yep: I ran this through DDT and verified that process0 gets file_system==ADIO_UFS/myerrcode==MPI_SUCCESS and process1 gets file_system=ADIO_NFS/myerrcode=MPI_SUCCESS. >> > > Thanks, Jeff. Big help pointing me to the crux of the problem. I don't know why I reduce the error code if it's not successful. We need to reduce the error code in all cases, and *then* reduce the detected file system type. > > Please try the patch below -- I don't have a mixed NFS/UFS environment I can try this out on. > > ==rob > > > > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > <0001-fix-fs-detection-when-multiple-fs-exist.patch>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpich-discuss] ROMIO filesystem check during MPI_File_open
by Rob Latham 24 Mar '14

24 Mar '14

On 03/24/2014 04:37 PM, Jeff Squyres (jsquyres) wrote: > On Mar 24, 2014, at 5:17 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > >> Every process will call ADIO_FileSysType_fncall() in the case where have_nfs_enabled==1 . Your interpretation of the code is so different from mine that I'm just going to have to past what I'm looking at and you can tell me where I'm wrong. These lines come from MPICH's ad_fstype.c. The content is the same in openmpi-1.6.4, just shifted by 25 or so lines: >> >> 637 ADIO_FileSysType_fncall(filename, &file_system, &myerrcode); >> 638 if (myerrcode != MPI_SUCCESS) { > > This is where our disconnect is occurring: _fncall returns myerrcode==MPI_SUCCESS in all cases. > > Meaning: the block with the Allreduce is not entered. This causes everyone to just use the file_system value they got from _fncall: one will have ADIO_UFS, and the rest will have ADIO_NFS. > > /Me checks again just to make sure I'm not goofing this up... Yep: I ran this through DDT and verified that process0 gets file_system==ADIO_UFS/myerrcode==MPI_SUCCESS and process1 gets file_system=ADIO_NFS/myerrcode=MPI_SUCCESS. > Thanks, Jeff. Big help pointing me to the crux of the problem. I don't know why I reduce the error code if it's not successful. We need to reduce the error code in all cases, and *then* reduce the detected file system type. Please try the patch below -- I don't have a mixed NFS/UFS environment I can try this out on. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] ROMIO filesystem check during MPI_File_open
by Jeff Squyres (jsquyres) 24 Mar '14

24 Mar '14

On Mar 24, 2014, at 5:17 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > Every process will call ADIO_FileSysType_fncall() in the case where have_nfs_enabled==1 . Your interpretation of the code is so different from mine that I'm just going to have to past what I'm looking at and you can tell me where I'm wrong. These lines come from MPICH's ad_fstype.c. The content is the same in openmpi-1.6.4, just shifted by 25 or so lines: > > 637 ADIO_FileSysType_fncall(filename, &file_system, &myerrcode); > 638 if (myerrcode != MPI_SUCCESS) { This is where our disconnect is occurring: _fncall returns myerrcode==MPI_SUCCESS in all cases. Meaning: the block with the Allreduce is not entered. This causes everyone to just use the file_system value they got from _fncall: one will have ADIO_UFS, and the rest will have ADIO_NFS. /Me checks again just to make sure I'm not goofing this up... Yep: I ran this through DDT and verified that process0 gets file_system==ADIO_UFS/myerrcode==MPI_SUCCESS and process1 gets file_system=ADIO_NFS/myerrcode=MPI_SUCCESS. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpich-discuss] ROMIO filesystem check during MPI_File_open
by Rob Latham 24 Mar '14

24 Mar '14

On 03/24/2014 02:49 PM, Jeff Squyres (jsquyres) wrote: > On Mar 14, 2014, at 11:23 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > >> I thought we handled this? we certianly seem to have made an effort: >> >> https://trac.mpich.org/projects/mpich/browser/src/mpi/romio/adio/common/ad_… > > Sorry for the delay in getting back to this... > > It looks like ADIO_FileSysType_fncall() is *not* collective -- it just does some magic to figure out what the local filesystem type is. that's correct. we expect _fncall to be called from one process in the "no nfs" case, and from every process in the "nfs enabled" case. > Then back up in ADIO_ResolveFileType(), that's called from two locations: MPI_File_open and MPI_File_delete. the only two functions that take a path and not an MPI_File handle. Open is collective, delete is not, but delete calls ResolveFileType with COMM_SELF (and you're not worried about the delete path anyway) > since have_nfs_enabled==1 and ADIO_FileSysType_fncall() returned MPI_SUCCESS, Every process will call ADIO_FileSysType_fncall() in the case where have_nfs_enabled==1 . Your interpretation of the code is so different from mine that I'm just going to have to past what I'm looking at and you can tell me where I'm wrong. These lines come from MPICH's ad_fstype.c. The content is the same in openmpi-1.6.4, just shifted by 25 or so lines: 637 ADIO_FileSysType_fncall(filename, &file_system, &myerrcode); 638 if (myerrcode != MPI_SUCCESS) { 639 *error_code = myerrcode; 640 [ giant comment omitted...] 651 652 MPI_Allreduce(error_code, &max_code, 1, MPI_INT, MPI_MAX, comm); 653 if (max_code != MPI_SUCCESS) { 654 *error_code = max_code; 655 return; 656 } 657 /* ensure everyone came up with the same file system type */ 658 MPI_Allreduce(&file_system, &min_code, 1, MPI_INT, 659 MPI_MIN, comm); 660 if (min_code == ADIO_NFS) file_system = ADIO_NFS; 661 } > MPI_Allreduce() is *not* called, and each process just proceeds with their local value for file_system. as you can see, there is an ALLREDUCE on the error code. Did anyone run into problems when we stat-ed the file system? If so, bail out early. If not, then let's figure out the common file system with a second ALLREDUCE. > That's where things go downhill: one process will get UFS, the rest will get NFS. Yeah, that would be a problem, and I'll gladly fix this code, but I can't see where we are failing. Am I looking in the right place? ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] ROMIO filesystem check during MPI_File_open
by Jeff Squyres (jsquyres) 24 Mar '14

24 Mar '14

On Mar 14, 2014, at 11:23 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > I thought we handled this? we certianly seem to have made an effort: > > https://trac.mpich.org/projects/mpich/browser/src/mpi/romio/adio/common/ad_… Sorry for the delay in getting back to this... It looks like ADIO_FileSysType_fncall() is *not* collective -- it just does some magic to figure out what the local filesystem type is. Then back up in ADIO_ResolveFileType(), since have_nfs_enabled==1 and ADIO_FileSysType_fncall() returned MPI_SUCCESS, MPI_Allreduce() is *not* called, and each process just proceeds with their local value for file_system. That's where things go downhill: one process will get UFS, the rest will get NFS. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpich-discuss] mpich-3.1 (stable release): make error.
by Jorge D'Elia 23 Mar '14

23 Mar '14

Dear Rajeev, Now "All 814 tests passed!" In the present case, since I was also doing an installation in a second cluster, I realized that there might be an inconsistency in the values of environment variables among server and nodes. Then, I go back to the first cluster, where the installation was carried out as root, and noticed that, in particular, the LD_LIBRARY_PATH values were different among server and nodes. Then, I made the configure (with all options) and make steps as an end user, with a consistent environment among server and nodes, and the "unknown size for unsigned int." error vanished. I attached the relevant files of the last successful installation. Thanks for your time and the links. Regards, Jorge. ----- Mensaje original ----- > De: "Rajeev Thakur" <thakur(a)mcs.anl.gov> > Para: discuss(a)mpich.org > Enviado: Domingo, 23 de Marzo 2014 13:26:10 > Asunto: Re: [mpich-discuss] mpich-3.1 (stable release): make error. > > There have been a few sporadic reports of the same error, > but I don't know what the resolution was. > > https://www.google.com/search?q=error:+%23error+%22unknown+size+for+unsigne… > > Rajeev > > On Mar 23, 2014, at 7:04 AM, Jorge D'Elia <jdelia(a)intec.unl.edu.ar> wrote: > > > > > Here I send the files obtained using only the prefix > > option (c.txt, m.txt and config.log). > > > > Regards, > > Jorge. > > > > ----- Mensaje original ----- > >> De: "Rajeev Thakur" <thakur(a)mcs.anl.gov> > >> Para: discuss(a)mpich.org > >> CC: mpich-discuss(a)mcs.anl.gov > >> Enviado: Sábado, 22 de Marzo 2014 23:46:25 > >> Asunto: Re: [mpich-discuss] mpich-3.1 (stable release): make error. > >> > >> Can you try configuring with just the --prefix option. > >> > >> Rajeev > >> > >> On Mar 22, 2014, at 12:01 PM, Jorge D'Elia <jdelia(a)intec.unl.edu.ar> > >> wrote: > >> > >>> Hi, > >>> > >>> I tried to install mpich-3.1 (stable release) but I find an > >>> error in the build step. I attached the files c.txt (configure > >>> step) and m.txt (make step with the VERBOSE=1 value). > >>> Please, any clue? Thanks in advance. Regards. Jorge. > >>> > >>> CIMEC (UNL-CONICET) Predio CONICET-Santa Fe, Colectora Ruta Nac 168, > >>> Paraje El Pozo, S3000GLN Santa Fe, ARGENTINA, http://www.cimec.org.ar/ > >>> Tel +54 342 451.15.94/95 ext 1018, fax: +54-342-451.11.69 > >>> txt>txt>_______________________________________________ > >>> discuss mailing list discuss(a)mpich.org > >>> To manage subscription options or unsubscribe: > >>> https://lists.mpich.org/mailman/listinfo/discuss > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > > <only-prefix.tgz>_______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] mpich-3.1 (stable release): make error.
by Rajeev Thakur 23 Mar '14

23 Mar '14

There have been a few sporadic reports of the same error, but I don't know what the resolution was. https://www.google.com/search?q=error:+%23error+%22unknown+size+for+unsigne… Rajeev On Mar 23, 2014, at 7:04 AM, Jorge D'Elia <jdelia(a)intec.unl.edu.ar> wrote: > > Here I send the files obtained using only the prefix > option (c.txt, m.txt and config.log). > > Regards, > Jorge. > > ----- Mensaje original ----- >> De: "Rajeev Thakur" <thakur(a)mcs.anl.gov> >> Para: discuss(a)mpich.org >> CC: mpich-discuss(a)mcs.anl.gov >> Enviado: Sábado, 22 de Marzo 2014 23:46:25 >> Asunto: Re: [mpich-discuss] mpich-3.1 (stable release): make error. >> >> Can you try configuring with just the --prefix option. >> >> Rajeev >> >> On Mar 22, 2014, at 12:01 PM, Jorge D'Elia <jdelia(a)intec.unl.edu.ar> >> wrote: >> >>> Hi, >>> >>> I tried to install mpich-3.1 (stable release) but I find an >>> error in the build step. I attached the files c.txt (configure >>> step) and m.txt (make step with the VERBOSE=1 value). >>> Please, any clue? Thanks in advance. Regards. Jorge. >>> >>> CIMEC (UNL-CONICET) Predio CONICET-Santa Fe, Colectora Ruta Nac 168, >>> Paraje El Pozo, S3000GLN Santa Fe, ARGENTINA, http://www.cimec.org.ar/ >>> Tel +54 342 451.15.94/95 ext 1018, fax: +54-342-451.11.69 >>> <c.txt><m.txt>_______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > <only-prefix.tgz>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0