- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI error, help needed
by Jayesh Krishna 28 Mar '13

28 Mar '13

Hi, The first error message indicates that the parameters specified to the program when running through VS is incorrect (Likely a VS configuration issue when running your code - I am assuming you are running your code as a singleton process here). In the second case (error message when running via command prompt), the error message indicates that you have specified a negative (or too large for an integer) count in the MPI_Scatter() call. The address of the receive buffer also looks to be NULL. You might have to debug your code or provide us more information so that we can help you. Regards, Jayesh ----- Original Message ----- From: "Suvarna Arakh" <arakhs(a)cwu.EDU> To: discuss(a)mpich.org Sent: Wednesday, March 27, 2013 2:48:54 PM Subject: [mpich-discuss] MPI error, help needed Hi, I am trying to run my MPI program on multiple processes, but I get the below errors, attached is the code. The code runs fine on 1 process. When running through Visual studio 2008: buffer provided too short for path: 4096 provided, 0 needed launch failed: CreateProcess(""ABC-HP c:\Users\ABC\Desktop\BS\Debug\BS.e xe) on 'ABC-HP' failed, error 87 - The parameter is incorrect. When running through command prompt: Fatal error in PMPI_Scatter: Invalid count, error stack: PMPI_Scatter(791): MPI_Scatter(sbuf=10030040, scount=-858993460, MPI_INT, rbuf=0 0000000, rcount=-858993460, MPI_INT, root=0, MPI_COMM_WORLD) failed PMPI_Scatter(728): Negative count, value is -858993460 Enter number of elements to be sorted job aborted: rank: node: exit code[: error message] 0: ABC-HP: 1: process 0 exited without calling finalize Please help. Regards. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI IO, reading MPI Writes without MPI read
by Rob Latham 28 Mar '13

28 Mar '13

On Thu, Mar 28, 2013 at 08:19:23AM -0700, Ryan Crocker wrote: > from the file those points are most definitely not in the same spot. We're using a set view so the memory is stored non-contiguously so you can't just read it back in, and i can see the offset in the data blocks. If you write the data with MPI_File_write_at, then you should read it back with MPI_File_read_at. Fortran I/O will expect the input to have various compiler-specific padding or ... well I don't properly know the details, other than it's a mess. ==rob > > On Mar 28, 2013, at 6:47 AM, Rajeev Thakur wrote: > > > Most MPI-IO implementations write plain binary files without any additional information. So the data should be exactly what you wrote and at the locations you specified. The disp is not stored in the file. It is just used to offset to the right location. The file won't be portable with the default "native" file format used for writing. In other words, it will be readable only on the same type of machine architecture. > > > > Try writing a simple MPI-IO program to write a file and then read it back using POSIX I/O calls (lseek, read) from a C program. I don't know if Fortran I/O expects files to be formatted in a particular way, which could cause a problem when reading from Fortran. > > > > Rajeev > > > > > > On Mar 28, 2013, at 2:00 AM, Ryan Crocker wrote: > > > >> So i'm not sure if this is crazy or not, but i have file outputs from my code that write ensight gold files in MPI. Here is the write, > >> > >> disp = 3*80+4+0*ncells_hexa*4 > >> call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_hexa,"native",mpi_info,ierr) > >> call MPI_FILE_WRITE_ALL(iunit,buffer3_hexa(:,1),ncells_hexa_,MPI_REAL_SP,status,ierr) > >> > >> if i wanted to read that in binary in fortran, or C (preferably fortran) what exactly would i need to do? I can't seem to write code that reads these in and produces anything that looks like the plot i get in paraview. I know that MPI write puts out each processor data vector with that disp in between them, but i just can't make that structure make sense to me when i try to read it in. > >> > >> Thanks for the help, > >> > >> Ryan Crocker > >> University of Vermont, School of Engineering > >> Mechanical Engineering Department > >> rcrocker(a)uvm.edu > >> 315-212-7331 > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > Ryan Crocker > University of Vermont, School of Engineering > Mechanical Engineering Department > rcrocker(a)uvm.edu > 315-212-7331 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] MPI IO, reading MPI Writes without MPI read
by Ryan Crocker 28 Mar '13

28 Mar '13

So write the vector to be written counter = 0 do k do j do i counter = counter +1 buffer(counter) = U(i,j,k) end do end do end do when i read that back in: counter = 0 do k do j do i counter = counter +1 U(i,j,k) = buffer(counter) end do end do end do from the file those points are most definitely not in the same spot. We're using a set view so the memory is stored non-contiguously so you can't just read it back in, and i can see the offset in the data blocks. On Mar 28, 2013, at 6:47 AM, Rajeev Thakur wrote: > Most MPI-IO implementations write plain binary files without any additional information. So the data should be exactly what you wrote and at the locations you specified. The disp is not stored in the file. It is just used to offset to the right location. The file won't be portable with the default "native" file format used for writing. In other words, it will be readable only on the same type of machine architecture. > > Try writing a simple MPI-IO program to write a file and then read it back using POSIX I/O calls (lseek, read) from a C program. I don't know if Fortran I/O expects files to be formatted in a particular way, which could cause a problem when reading from Fortran. > > Rajeev > > > On Mar 28, 2013, at 2:00 AM, Ryan Crocker wrote: > >> So i'm not sure if this is crazy or not, but i have file outputs from my code that write ensight gold files in MPI. Here is the write, >> >> disp = 3*80+4+0*ncells_hexa*4 >> call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_hexa,"native",mpi_info,ierr) >> call MPI_FILE_WRITE_ALL(iunit,buffer3_hexa(:,1),ncells_hexa_,MPI_REAL_SP,status,ierr) >> >> if i wanted to read that in binary in fortran, or C (preferably fortran) what exactly would i need to do? I can't seem to write code that reads these in and produces anything that looks like the plot i get in paraview. I know that MPI write puts out each processor data vector with that disp in between them, but i just can't make that structure make sense to me when i try to read it in. >> >> Thanks for the help, >> >> Ryan Crocker >> University of Vermont, School of Engineering >> Mechanical Engineering Department >> rcrocker(a)uvm.edu >> 315-212-7331 >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss Ryan Crocker University of Vermont, School of Engineering Mechanical Engineering Department rcrocker(a)uvm.edu 315-212-7331

1 0

Re: [mpich-discuss] MPI IO, reading MPI Writes without MPI read
by Rajeev Thakur 28 Mar '13

28 Mar '13

Most MPI-IO implementations write plain binary files without any additional information. So the data should be exactly what you wrote and at the locations you specified. The disp is not stored in the file. It is just used to offset to the right location. The file won't be portable with the default "native" file format used for writing. In other words, it will be readable only on the same type of machine architecture. Try writing a simple MPI-IO program to write a file and then read it back using POSIX I/O calls (lseek, read) from a C program. I don't know if Fortran I/O expects files to be formatted in a particular way, which could cause a problem when reading from Fortran. Rajeev On Mar 28, 2013, at 2:00 AM, Ryan Crocker wrote: > So i'm not sure if this is crazy or not, but i have file outputs from my code that write ensight gold files in MPI. Here is the write, > > disp = 3*80+4+0*ncells_hexa*4 > call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_hexa,"native",mpi_info,ierr) > call MPI_FILE_WRITE_ALL(iunit,buffer3_hexa(:,1),ncells_hexa_,MPI_REAL_SP,status,ierr) > > if i wanted to read that in binary in fortran, or C (preferably fortran) what exactly would i need to do? I can't seem to write code that reads these in and produces anything that looks like the plot i get in paraview. I know that MPI write puts out each processor data vector with that disp in between them, but i just can't make that structure make sense to me when i try to read it in. > > Thanks for the help, > > Ryan Crocker > University of Vermont, School of Engineering > Mechanical Engineering Department > rcrocker(a)uvm.edu > 315-212-7331 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI IO, reading MPI Writes without MPI read
by Ryan Crocker 28 Mar '13

28 Mar '13

Here is the whole code snippet for writing a vector, it will probably help: inquire(file=file,exist=file_is_there) if (file_is_there .and. irank.eq.iroot) call MPI_FILE_DELETE(file,mpi_info,ierr) call MPI_FILE_OPEN(comm,file,IOR(MPI_MODE_WRONLY,MPI_MODE_CREATE),mpi_info,iunit,ierr) ! Write header (only root) if (irank.eq.iroot) then buffer = trim(adjustl(name)) size = 80 call MPI_FILE_WRITE(iunit,buffer,size,MPI_CHARACTER,status,ierr) buffer = 'part' size = 80 call MPI_FILE_WRITE(iunit,buffer,size,MPI_CHARACTER,status,ierr) ibuffer = 1 size = 1 call MPI_FILE_WRITE(iunit,ibuffer,size,MPI_INTEGER,status,ierr) buffer = 'hexa8' size = 80 call MPI_FILE_WRITE(iunit,buffer,size,MPI_CHARACTER,status,ierr) end if ! Write the data disp = 3*80+4+0*ncells_hexa*4 call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_hexa,"native",mpi_info,ierr) call MPI_FILE_WRITE_ALL(iunit,buffer3_hexa(:,1),ncells_hexa_,MPI_REAL_SP,status,ierr) disp = 3*80+4+1*ncells_hexa*4 call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_hexa,"native",mpi_info,ierr) call MPI_FILE_WRITE_ALL(iunit,buffer3_hexa(:,2),ncells_hexa_,MPI_REAL_SP,status,ierr) disp = 3*80+4+2*ncells_hexa*4 call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_hexa,"native",mpi_info,ierr) call MPI_FILE_WRITE_ALL(iunit,buffer3_hexa(:,3),ncells_hexa_,MPI_REAL_SP,status,ierr) ! Write the file - wedge if (ncells_wedge.gt.0) then disp = 3*80+4+3*ncells_hexa*4+0*ncells_wedge*4 call MPI_FILE_SET_VIEW(iunit,disp,MPI_CHARACTER,MPI_CHARACTER,"native",mpi_info,ierr) if (irank.eq.iroot) then buffer = 'penta6' size = 80 call MPI_FILE_WRITE(iunit,buffer,size,MPI_CHARACTER,status,ierr) end if disp = 3*80+4+3*ncells_hexa*4+80+0*ncells_wedge*4 call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_wedge,"native",mpi_info,ierr) call MPI_FILE_WRITE_ALL(iunit,buffer3_wedge(:,1),ncells_wedge_,MPI_REAL_SP,status,ierr) disp = 3*80+4+3*ncells_hexa*4+80+1*ncells_wedge*4 call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_wedge,"native",mpi_info,ierr) call MPI_FILE_WRITE_ALL(iunit,buffer3_wedge(:,2),ncells_wedge_,MPI_REAL_SP,status,ierr) disp = 3*80+4+3*ncells_hexa*4+80+2*ncells_wedge*4 call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_wedge,"native",mpi_info,ierr) call MPI_FILE_WRITE_ALL(iunit,buffer3_wedge(:,3),ncells_wedge_,MPI_REAL_SP,status,ierr) end if ! Close the file call MPI_FILE_CLOSE(iunit,ierr) On Mar 28, 2013, at 12:00 AM, Ryan Crocker wrote: > So i'm not sure if this is crazy or not, but i have file outputs from my code that write ensight gold files in MPI. Here is the write, > > disp = 3*80+4+0*ncells_hexa*4 > call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_hexa,"native",mpi_info,ierr) > call MPI_FILE_WRITE_ALL(iunit,buffer3_hexa(:,1),ncells_hexa_,MPI_REAL_SP,status,ierr) > > if i wanted to read that in binary in fortran, or C (preferably fortran) what exactly would i need to do? I can't seem to write code that reads these in and produces anything that looks like the plot i get in paraview. I know that MPI write puts out each processor data vector with that disp in between them, but i just can't make that structure make sense to me when i try to read it in. > > Thanks for the help, > > Ryan Crocker > University of Vermont, School of Engineering > Mechanical Engineering Department > rcrocker(a)uvm.edu > 315-212-7331 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss Ryan Crocker University of Vermont, School of Engineering Mechanical Engineering Department rcrocker(a)uvm.edu 315-212-7331

1 0

MPI IO, reading MPI Writes without MPI read
by Ryan Crocker 28 Mar '13

28 Mar '13

So i'm not sure if this is crazy or not, but i have file outputs from my code that write ensight gold files in MPI. Here is the write, disp = 3*80+4+0*ncells_hexa*4 call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_hexa,"native",mpi_info,ierr) call MPI_FILE_WRITE_ALL(iunit,buffer3_hexa(:,1),ncells_hexa_,MPI_REAL_SP,status,ierr) if i wanted to read that in binary in fortran, or C (preferably fortran) what exactly would i need to do? I can't seem to write code that reads these in and produces anything that looks like the plot i get in paraview. I know that MPI write puts out each processor data vector with that disp in between them, but i just can't make that structure make sense to me when i try to read it in. Thanks for the help, Ryan Crocker University of Vermont, School of Engineering Mechanical Engineering Department rcrocker(a)uvm.edu 315-212-7331

1 0

MPI error, help needed
by Suvarna Arakh 27 Mar '13

27 Mar '13

Hi, I am trying to run my MPI program on multiple processes, but I get the below errors, attached is the code. The code runs fine on 1 process. When running through Visual studio 2008: buffer provided too short for path: 4096 provided, 0 needed launch failed: CreateProcess(""ABC-HP c:\Users\ABC\Desktop\BS\Debug\BS.e xe) on 'ABC-HP' failed, error 87 - The parameter is incorrect. When running through command prompt: Fatal error in PMPI_Scatter: Invalid count, error stack: PMPI_Scatter(791): MPI_Scatter(sbuf=10030040, scount=-858993460, MPI_INT, rbuf=0 0000000, rcount=-858993460, MPI_INT, root=0, MPI_COMM_WORLD) failed PMPI_Scatter(728): Negative count, value is -858993460 Enter number of elements to be sorted job aborted: rank: node: exit code[: error message] 0: ABC-HP: 1: process 0 exited without calling finalize Please help. Regards.

1 0

Re: [mpich-discuss] Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Pavan Balaji 26 Mar '13

26 Mar '13

Hi Tim, I didn't mean for you to give me a 1-paragraph summary of the communication model. My first guess is a neighborhood collective (in MPI-3), which Jeff already suggested, but you'll need to look through the capabilities of the different MPI operations and decide. With respect to MPI implementations, I guess most MPICH derivatives (including MVAPICH, Cray MPI, IBM MPI, Intel MPI, etc.) should have MPI-3 capability this year or early next year, though we don't have an official date from most of them. I believe the latest release of MVAPICH already has MPI-3 support. -- Pavan On 03/26/2013 01:46 PM US Central Time, Timothy Stitt wrote: > Pavan...thanks for the comments. > > Our finite element simulation decomposes the problem domain into a 2D grid of sub-domains with a 1-1 mapping between compute cores and sub-domains. In each sub-domain we apply a ghost zone between direct neighbors and communicate the relevant data between neighbors during each timestep. > > In general we use MVAPICH2 on the various systems we have access to. When using a Cray we use their tuned MPI library which I believe is derived from MPICH2. If we primarily are MPICH-based (and with a basic idea of our communication pattern as given above) would you recommend looking into some other MPI(-3) functionality to achieve better scalability and performance? > > Cheers, > > Tim. > > On Mar 26, 2013, at 12:59 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> Tim, >> >> It's hard for us to decide which MPI functionality is better for you >> without knowing your algorithmic model. For some algorithms persistent >> send/recv is great. For some, RMA is great. Of course, each one has >> its shortcomings. Please don't jump to one or the other based on a >> short email discussion :-). >> >> Also note that some MPI implementations do optimize persistent >> communication, particularly for memory registration. So even if you >> don't see a benefit on some platforms, it doesn't mean that other MPI >> implementations cannot take advantage of it. >> >> -- Pavan >> >> On 03/26/2013 11:36 AM US Central Time, Timothy Stitt wrote: >>> Thanks for the quick reply Jeff. That information is valuable. I'll >>> follow up on your pointers. >>> >>> Much appreciated, >>> >>> Tim. >>> >>> On Mar 26, 2013, at 12:32 PM, Jeff Hammond <jhammond(a)alcf.anl.gov >>> <mailto:[email protected]>> wrote: >>> >>>> You might want to look at neighborhood collectives, which are >>>> discussed in Chapter 7 of MPI-3. This is a new feature so it may not >>>> be implemented in all MPI implementations, but MPICH supports it. I >>>> guess MVAPICH will support it soon enough if not already. >>>> >>>> When persistent MPI send/recv is discussed at the MPI Forum, it is >>>> often described as an inadequate solution because it does not specify >>>> a full channel and thus some important optimizations, e.g. for RDMA, >>>> may not be feasible. >>>> >>>> If you can use MPI-3 RMA, that is probably going to be a good idea, >>>> although high-quality support for RMA varies. MPICH-derived >>>> implementations usually do a good job though. >>>> >>>> Best, >>>> >>>> Jeff >>>> >>>> On Tue, Mar 26, 2013 at 11:25 AM, Timothy Stitt >>>> <Timothy.Stitt.9(a)nd.edu <mailto:[email protected]>> wrote: >>>>> Hi all, >>>>> >>>>> I've been asking this question around various MPI boards to try and >>>>> get a consensus before I decide to rewrite some MPI code. I am >>>>> grateful for any advice that you can give. >>>>> >>>>> I've inherited a MPI code that was written ~8-10 years ago and it >>>>> predominately uses MPI persistent communication routines for data >>>>> transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I >>>>> am not familiar with and don't normally hear much discussion about). >>>>> I was just wondering if using persistent communication calls is still >>>>> regarded as the most efficient/scalable way to perform communication >>>>> when the communication pattern is known and fixed amongst >>>>> neighborhood processes? We regularly run the code across an IB >>>>> network so would there be a benefit to rewrite the code using another >>>>> approach (e.g. MPI one-sided communication) or should I leave it as >>>>> it is? The code currently scales up to 10K cores and I want to push >>>>> it even further and thus was wondering if there is any benefit in >>>>> tinkering with this persistent MPI communication approach? >>>>> >>>>> Thanks in advance for any advice. >>>>> >>>>> Tim. >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> Argonne Leadership Computing Facility >>>> University of Chicago Computation Institute >>>> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 >>>> http://www.linkedin.com/in/jeffhammond >>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Timothy Stitt 26 Mar '13

26 Mar '13

Pavan...thanks for the comments. Our finite element simulation decomposes the problem domain into a 2D grid of sub-domains with a 1-1 mapping between compute cores and sub-domains. In each sub-domain we apply a ghost zone between direct neighbors and communicate the relevant data between neighbors during each timestep. In general we use MVAPICH2 on the various systems we have access to. When using a Cray we use their tuned MPI library which I believe is derived from MPICH2. If we primarily are MPICH-based (and with a basic idea of our communication pattern as given above) would you recommend looking into some other MPI(-3) functionality to achieve better scalability and performance? Cheers, Tim. On Mar 26, 2013, at 12:59 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > Tim, > > It's hard for us to decide which MPI functionality is better for you > without knowing your algorithmic model. For some algorithms persistent > send/recv is great. For some, RMA is great. Of course, each one has > its shortcomings. Please don't jump to one or the other based on a > short email discussion :-). > > Also note that some MPI implementations do optimize persistent > communication, particularly for memory registration. So even if you > don't see a benefit on some platforms, it doesn't mean that other MPI > implementations cannot take advantage of it. > > -- Pavan > > On 03/26/2013 11:36 AM US Central Time, Timothy Stitt wrote: >> Thanks for the quick reply Jeff. That information is valuable. I'll >> follow up on your pointers. >> >> Much appreciated, >> >> Tim. >> >> On Mar 26, 2013, at 12:32 PM, Jeff Hammond <jhammond(a)alcf.anl.gov >> <mailto:[email protected]>> wrote: >> >>> You might want to look at neighborhood collectives, which are >>> discussed in Chapter 7 of MPI-3. This is a new feature so it may not >>> be implemented in all MPI implementations, but MPICH supports it. I >>> guess MVAPICH will support it soon enough if not already. >>> >>> When persistent MPI send/recv is discussed at the MPI Forum, it is >>> often described as an inadequate solution because it does not specify >>> a full channel and thus some important optimizations, e.g. for RDMA, >>> may not be feasible. >>> >>> If you can use MPI-3 RMA, that is probably going to be a good idea, >>> although high-quality support for RMA varies. MPICH-derived >>> implementations usually do a good job though. >>> >>> Best, >>> >>> Jeff >>> >>> On Tue, Mar 26, 2013 at 11:25 AM, Timothy Stitt >>> <Timothy.Stitt.9(a)nd.edu <mailto:[email protected]>> wrote: >>>> Hi all, >>>> >>>> I've been asking this question around various MPI boards to try and >>>> get a consensus before I decide to rewrite some MPI code. I am >>>> grateful for any advice that you can give. >>>> >>>> I've inherited a MPI code that was written ~8-10 years ago and it >>>> predominately uses MPI persistent communication routines for data >>>> transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I >>>> am not familiar with and don't normally hear much discussion about). >>>> I was just wondering if using persistent communication calls is still >>>> regarded as the most efficient/scalable way to perform communication >>>> when the communication pattern is known and fixed amongst >>>> neighborhood processes? We regularly run the code across an IB >>>> network so would there be a benefit to rewrite the code using another >>>> approach (e.g. MPI one-sided communication) or should I leave it as >>>> it is? The code currently scales up to 10K cores and I want to push >>>> it even further and thus was wondering if there is any benefit in >>>> tinkering with this persistent MPI communication approach? >>>> >>>> Thanks in advance for any advice. >>>> >>>> Tim. >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> -- >>> Jeff Hammond >>> Argonne Leadership Computing Facility >>> University of Chicago Computation Institute >>> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 >>> http://www.linkedin.com/in/jeffhammond >>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Pavan Balaji 26 Mar '13

26 Mar '13

Tim, It's hard for us to decide which MPI functionality is better for you without knowing your algorithmic model. For some algorithms persistent send/recv is great. For some, RMA is great. Of course, each one has its shortcomings. Please don't jump to one or the other based on a short email discussion :-). Also note that some MPI implementations do optimize persistent communication, particularly for memory registration. So even if you don't see a benefit on some platforms, it doesn't mean that other MPI implementations cannot take advantage of it. -- Pavan On 03/26/2013 11:36 AM US Central Time, Timothy Stitt wrote: > Thanks for the quick reply Jeff. That information is valuable. I'll > follow up on your pointers. > > Much appreciated, > > Tim. > > On Mar 26, 2013, at 12:32 PM, Jeff Hammond <jhammond(a)alcf.anl.gov > <mailto:[email protected]>> wrote: > >> You might want to look at neighborhood collectives, which are >> discussed in Chapter 7 of MPI-3. This is a new feature so it may not >> be implemented in all MPI implementations, but MPICH supports it. I >> guess MVAPICH will support it soon enough if not already. >> >> When persistent MPI send/recv is discussed at the MPI Forum, it is >> often described as an inadequate solution because it does not specify >> a full channel and thus some important optimizations, e.g. for RDMA, >> may not be feasible. >> >> If you can use MPI-3 RMA, that is probably going to be a good idea, >> although high-quality support for RMA varies. MPICH-derived >> implementations usually do a good job though. >> >> Best, >> >> Jeff >> >> On Tue, Mar 26, 2013 at 11:25 AM, Timothy Stitt >> <Timothy.Stitt.9(a)nd.edu <mailto:[email protected]>> wrote: >>> Hi all, >>> >>> I've been asking this question around various MPI boards to try and >>> get a consensus before I decide to rewrite some MPI code. I am >>> grateful for any advice that you can give. >>> >>> I've inherited a MPI code that was written ~8-10 years ago and it >>> predominately uses MPI persistent communication routines for data >>> transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I >>> am not familiar with and don't normally hear much discussion about). >>> I was just wondering if using persistent communication calls is still >>> regarded as the most efficient/scalable way to perform communication >>> when the communication pattern is known and fixed amongst >>> neighborhood processes? We regularly run the code across an IB >>> network so would there be a benefit to rewrite the code using another >>> approach (e.g. MPI one-sided communication) or should I leave it as >>> it is? The code currently scales up to 10K cores and I want to push >>> it even further and thus was wondering if there is any benefit in >>> tinkering with this persistent MPI communication approach? >>> >>> Thanks in advance for any advice. >>> >>> Tim. >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > *Tim Stitt* PhD > User Support Manager (CRC) > Research Assistant Professor (Computer Science & Engineering) > Room 108, Center for Research Computing | University of Notre Dame | > P.O. Box 539, Notre Dame, IN 46556 | Phone: 574-631-5287 | Email: > tstitt(a)nd.edu <mailto:[email protected]> > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0