April 2015 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] How to use non-blocking send/receive without calling MPI_Wait
by Jeff Hammond 07 Apr '15

07 Apr '15

I don't expect MPI to make async progress on Ethernet unless you have a comm thread in MPI poking the network. In MPICH, you would use MPICH_ASYNC_PROGRESS=1. There is something similar in Intel MPI, but I do not know the same of the environment variable (this may be ironic, given I work for Intel, but such is life). Your solution with OpenMP of dedicating a comm thread makes sense. I'd make thread 0 the comm thread and let the rest of the threads compute, since this might have some advantages relative to NUNA or thread migration, but it's probably not a significant effect on most platforms. Also, this is technically required for using MPI_THREAD_FUNNELED, but I know of no implementation where it actually matters. At this point, I don't know what more you want. Your expectation of perfect overlap is not justified on Ethernet and you appear to have written reasonable code for your problem of interest. Jeff On Tue, Apr 7, 2015 at 12:48 AM, Lei Shi <lshi(a)ku.edu> wrote: > Here is my pure MPI overlap version. I use intel traceanalyzer, the > profiling shows that right now, communication only proceed when I call > mpi_waitall on nodes with 10g network. > > /** pure mpi overlap **/ > template<typename T> > void CPR_NS_3D_Solver<T>::UpdateRes(T**q, T**res){ > if(_n_proc>1) > SendInterfaceSol(); //call isend/irecv to send msg 1 > > ResFromDivInvisFlux(q,res); //do local jobs > > if(_n_proc>1){ > RevInterfaceSol(); //mpi_waitall for msg 1 > if(vis_mode_) > SendInterfaceCorrGrad(); //depends on msg 1 then snd msg 2 > } > > if(vis_mode_) > ResFromDivVisFlux(q,res); //computing, which depends on msg 1 > > if(_n_proc>1 && vis_mode_) > RevInterfaceCorrGrad(); //mpi_waitall for msg 2 > > ResFromFluxCorrection(q,res); //computing, which depends on msg 1 and 2 > } > > > On Tue, Apr 7, 2015 at 2:39 AM, Lei Shi <lshi(a)ku.edu> wrote: > >> >> >> On Tue, Apr 7, 2015 at 2:37 AM, Lei Shi <leishi(a)ku.edu> wrote: >> >>> Hi Huiwei and Jeff, >>> >>> I use hybrid OpenMP/MPI to do overlap communication. So I put all >>> communication in one dedicated OpenMP thread and computation in the other >>> thread. For this case, I'm using intel MPI library. Probably I did some >>> mistakes >>> >>> One version of my code using one dedicated thread to do messaging is >>> like this >>> >>> /* hybrid mpi/openmp overlap **/template<typename T>void CPR_NS_3D_Solver<T>::UpdateRes(T**q, T**res){ >>> int thread_id,n_thread; >>> int sol_rev_flag=0,grad_rev_flag=0; >>> >>> // Explicitly disable dynamic teams >>> omp_set_dynamic(0); >>> // Use 2 threads for all consecutive parallel regions >>> omp_set_num_threads(2); >>> #pragma omp parallel default(shared) private(thread_id) >>> { >>> thread_id=omp_get_thread_num(); >>> n_thread=omp_get_num_threads(); >>> >>> /** communication thread **/ >>> if(thread_id==1){ >>> SendInterfaceSol(); >>> RevInterfaceSol();#pragma omp flush >>> sol_rev_flag=1;#pragma omp flush(sol_rev_flag) >>> } >>> >>> /** computation thread **/ >>> if(thread_id==0){ >>> ResFromDivInvisFlux(q,res); //local computation >>> #pragma omp flush(sol_rev_flag) >>> while(sol_rev_flag!=1){ #pragma omp flush(sol_rev_flag) >>> }#pragma omp flush >>> ResFromFluxCorrection(q,res); //depends on interface sol >>> } >>> }//end of omp >>> } >>> >>> template<typename T> >>> >>> void CPR_NS_3D_Solver<T>::SendInterfaceSol(){ >>> uint *n_if_to_proc=this->grid_->num_iface_proc; >>> uint **if_to_proc=this->grid_->snd_iface_proc; >>> uint **rev_if_to_f=this->grid_->rev_iface_proc; >>> >>> int tag=52; >>> for(int p2=0;p2<_n_proc;++p2){ >>> if(p2!=_proc_id){ >>> int nif=n_if_to_proc[p2]; >>> //pack data to send .... >>> >>> } >>> } >>> >>> /** * Exchange interface sol **/ >>> int n_proc_exchange=0; >>> for(int z=0;z<_n_proc;++z){ >>> int nif=n_if_to_proc[z]; >>> >>> //send data >>> if(nif>0){ MPI_Isend(&snd_buf_[z][0],n_buf_[z],MPI_DOUBLE,z,tag, MPI_COMM_WORLD, &s_sol_req_[n_proc_exchange]); >>> MPI_Irecv(&rev_buf_[z][0],n_buf_[z],MPI_DOUBLE,z,tag, MPI_COMM_WORLD, &r_sol_req_[n_proc_exchange]); >>> n_proc_exchange++; >>> } >>> } >>> >>> } >>> >>> template<typename T> >>> void CPR_NS_3D_Solver<T>::RevInterfaceSol(){ uint *n_if_to_proc=this->grid_->num_iface_proc; >>> uint **if_to_proc=this->grid_->snd_iface_proc; >>> uint **rev_if_to_f=this->grid_->rev_iface_proc; >>> >>> //wait >>> if(n_proc_exchange_>0){ MPI_Waitall(n_proc_exchange_,s_sol_req_,MPI_STATUS_IGNORE); MPI_Waitall(n_proc_exchange_,r_sol_req_,MPI_STATUS_IGNORE); } >>> >>> /** store to local data structure **/ >>> for(int z=0;z<_n_proc;++z){ >>> int nif=n_if_to_proc[z]; >>> >>> if(nif>0){ >>> >>> //unpacking .... } >>> } >>> >>> } >>> >>> >>> >>> >>> >>> >>> Sincerely Yours, >>> >>> Lei Shi >>> --------- >>> >>> On Fri, Apr 3, 2015 at 4:37 PM, Jeff Hammond <jeff.science(a)gmail.com> >>> wrote: >>> >>>> As far as I know, Ethernet is not good at making asynchronous progress >>>> in hardware the way e.g. InfiniBand is. I would have thought that a >>>> dedicated progress thread would help, but it seems you tried that. Did you >>>> use your own progress thread or MPICH_ASYNC_PROGRESS=1? >>>> >>>> Jeff >>>> >>>> On Fri, Apr 3, 2015 at 10:10 AM, Lei Shi <lshi(a)ku.edu> wrote: >>>> >>>>> Huiwei, >>>>> >>>>> Thanks for your email. Your answer leads to my another question about >>>>> asynchronous MPI communication. >>>>> >>>>> I'm trying to do an overlapped communication/computing to speedup my >>>>> MPI code. I read some papers comparing some different approaches to do the >>>>> overlapped communication. The "naive" overlapped communication >>>>> implementation, which only use non-blocking mpi Isend/Irecv and the hybrid >>>>> approach using OpenMP and MPI together. In the hybrid approach, a separated >>>>> thread is use to do all non-blocking communications. Just exactly as you >>>>> said, the results indicate that current MPI implementations do not >>>>> support true asynchronous communication. >>>>> >>>>> If I use the naive approach, my code with non-blocking or blocking >>>>> send/recv gives me almost the same performance in term of Wtime. All >>>>> communications are postponed to MPI_Wait. >>>>> >>>>> I have tried calling mpi_test to push library to do communication >>>>> during iterations. And try to use a dedicated thread to do communication >>>>> and the other thread to do computing only. However, the performance gains >>>>> are very small or no gain at all. I'm wondering it is due to the hardware. >>>>> The cluster I tested uses 10G Ethernet card. >>>>> >>>>> >>>>> Best, >>>>> >>>>> Lei Shi >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Apr 3, 2015 at 8:49 AM, Huiwei Lu <huiweilu(a)mcs.anl.gov> >>>>> wrote: >>>>> >>>>>> Hi Lei, >>>>>> >>>>>> As far as I know, all current MPI implementations do not support true >>>>>> asynchronous communication for now. i.e., If there is no MPI calls in your >>>>>> iterations, MPICH will not be able to make progress on communication. >>>>>> >>>>>> One solution is to poll the MPI runtime regularly to make progress by >>>>>> inserting MPI_Test to your iteration (even though you do not want to check >>>>>> the data). >>>>>> >>>>>> Another solution is to enable MPI's asynchronous progress thread to >>>>>> make progress for you. >>>>>> >>>>>> -- >>>>>> Huiwei >>>>>> >>>>>> On Thu, Apr 2, 2015 at 11:44 PM, Lei Shi <lshi(a)ku.edu> wrote: >>>>>> >>>>>>> Hi Junchao, >>>>>>> >>>>>>> Thanks for your reply. For my case, I don't want to check the data >>>>>>> have been received or not. So I don't want to call MPI_Test or any function >>>>>>> to verify it. But my problem is like if I ignore calling the MPI_Wait, just >>>>>>> call Isend/Irev, my program freezes for several sec and then continues to >>>>>>> run. My guess is probably I messed up the MPI library internal buffer by >>>>>>> doing this. >>>>>>> >>>>>>> On Thu, Apr 2, 2015 at 7:25 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> >>>>>>> wrote: >>>>>>> >>>>>>>> Does MPI_Test fit your needs? >>>>>>>> >>>>>>>> --Junchao Zhang >>>>>>>> >>>>>>>> On Thu, Apr 2, 2015 at 7:16 PM, Lei Shi <lshi(a)ku.edu> wrote: >>>>>>>> >>>>>>>>> I want to use non-blocking send/rev MPI_Isend/MPI_Irev to do >>>>>>>>> communication. But in my case, I don't really care what kind of data I get >>>>>>>>> or it is ready to use or not. So I don't want to waste my time to do any >>>>>>>>> synchronization by calling MPI_Wait or etc API. >>>>>>>>> >>>>>>>>> But when I avoid calling MPI_Wait, my program is freezed several >>>>>>>>> secs after running some iterations (after multiple MPI_Isend/Irev >>>>>>>>> callings), then continues. It takes even more time than the case with >>>>>>>>> MPI_Wait. So my question is how to do a "true" non-blocking communication >>>>>>>>> without waiting for the data ready or not. Thanks. >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>>>> To manage subscription options or unsubscribe: >>>>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>>> To manage subscription options or unsubscribe: >>>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> jeff.science(a)gmail.com >>>> http://jeffhammond.github.io/ >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >>> >> > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to use non-blocking send/receive without calling MPI_Wait
by Huiwei Lu 07 Apr '15

07 Apr '15

Hi Lei, Your profiling is correct. MPI_Isend/Irecv only start the communication but the real progress is made in MPI_Waitall. If you want to overlap communication with ResFromDivInvisFlux, you can put MPI_Test inside ResFromDivInvisFlux to force MPI to make progress periodically when MPI_Test is called. Another option is to use asynchronous progress thread. Check MPIR_CVAR_ASYNC_PROGRESS for more information. Best regards, -- Huiwei On Tue, Apr 7, 2015 at 2:48 AM, Lei Shi <lshi(a)ku.edu> wrote: > Here is my pure MPI overlap version. I use intel traceanalyzer, the > profiling shows that right now, communication only proceed when I call > mpi_waitall on nodes with 10g network. > > /** pure mpi overlap **/ > template<typename T> > void CPR_NS_3D_Solver<T>::UpdateRes(T**q, T**res){ > if(_n_proc>1) > SendInterfaceSol(); //call isend/irecv to send msg 1 > > ResFromDivInvisFlux(q,res); //do local jobs > > if(_n_proc>1){ > RevInterfaceSol(); //mpi_waitall for msg 1 > if(vis_mode_) > SendInterfaceCorrGrad(); //depends on msg 1 then snd msg 2 > } > > if(vis_mode_) > ResFromDivVisFlux(q,res); //computing, which depends on msg 1 > > if(_n_proc>1 && vis_mode_) > RevInterfaceCorrGrad(); //mpi_waitall for msg 2 > > ResFromFluxCorrection(q,res); //computing, which depends on msg 1 and 2 > } > > > On Tue, Apr 7, 2015 at 2:39 AM, Lei Shi <lshi(a)ku.edu> wrote: > >> >> >> On Tue, Apr 7, 2015 at 2:37 AM, Lei Shi <leishi(a)ku.edu> wrote: >> >>> Hi Huiwei and Jeff, >>> >>> I use hybrid OpenMP/MPI to do overlap communication. So I put all >>> communication in one dedicated OpenMP thread and computation in the other >>> thread. For this case, I'm using intel MPI library. Probably I did some >>> mistakes >>> >>> One version of my code using one dedicated thread to do messaging is >>> like this >>> >>> /* hybrid mpi/openmp overlap **/template<typename T>void CPR_NS_3D_Solver<T>::UpdateRes(T**q, T**res){ >>> int thread_id,n_thread; >>> int sol_rev_flag=0,grad_rev_flag=0; >>> >>> // Explicitly disable dynamic teams >>> omp_set_dynamic(0); >>> // Use 2 threads for all consecutive parallel regions >>> omp_set_num_threads(2); >>> #pragma omp parallel default(shared) private(thread_id) >>> { >>> thread_id=omp_get_thread_num(); >>> n_thread=omp_get_num_threads(); >>> >>> /** communication thread **/ >>> if(thread_id==1){ >>> SendInterfaceSol(); >>> RevInterfaceSol();#pragma omp flush >>> sol_rev_flag=1;#pragma omp flush(sol_rev_flag) >>> } >>> >>> /** computation thread **/ >>> if(thread_id==0){ >>> ResFromDivInvisFlux(q,res); //local computation >>> #pragma omp flush(sol_rev_flag) >>> while(sol_rev_flag!=1){ #pragma omp flush(sol_rev_flag) >>> }#pragma omp flush >>> ResFromFluxCorrection(q,res); //depends on interface sol >>> } >>> }//end of omp >>> } >>> >>> template<typename T> >>> >>> void CPR_NS_3D_Solver<T>::SendInterfaceSol(){ >>> uint *n_if_to_proc=this->grid_->num_iface_proc; >>> uint **if_to_proc=this->grid_->snd_iface_proc; >>> uint **rev_if_to_f=this->grid_->rev_iface_proc; >>> >>> int tag=52; >>> for(int p2=0;p2<_n_proc;++p2){ >>> if(p2!=_proc_id){ >>> int nif=n_if_to_proc[p2]; >>> //pack data to send .... >>> >>> } >>> } >>> >>> /** * Exchange interface sol **/ >>> int n_proc_exchange=0; >>> for(int z=0;z<_n_proc;++z){ >>> int nif=n_if_to_proc[z]; >>> >>> //send data >>> if(nif>0){ MPI_Isend(&snd_buf_[z][0],n_buf_[z],MPI_DOUBLE,z,tag, MPI_COMM_WORLD, &s_sol_req_[n_proc_exchange]); >>> MPI_Irecv(&rev_buf_[z][0],n_buf_[z],MPI_DOUBLE,z,tag, MPI_COMM_WORLD, &r_sol_req_[n_proc_exchange]); >>> n_proc_exchange++; >>> } >>> } >>> >>> } >>> >>> template<typename T> >>> void CPR_NS_3D_Solver<T>::RevInterfaceSol(){ uint *n_if_to_proc=this->grid_->num_iface_proc; >>> uint **if_to_proc=this->grid_->snd_iface_proc; >>> uint **rev_if_to_f=this->grid_->rev_iface_proc; >>> >>> //wait >>> if(n_proc_exchange_>0){ MPI_Waitall(n_proc_exchange_,s_sol_req_,MPI_STATUS_IGNORE); MPI_Waitall(n_proc_exchange_,r_sol_req_,MPI_STATUS_IGNORE); } >>> >>> /** store to local data structure **/ >>> for(int z=0;z<_n_proc;++z){ >>> int nif=n_if_to_proc[z]; >>> >>> if(nif>0){ >>> >>> //unpacking .... } >>> } >>> >>> } >>> >>> >>> >>> >>> >>> >>> Sincerely Yours, >>> >>> Lei Shi >>> --------- >>> >>> On Fri, Apr 3, 2015 at 4:37 PM, Jeff Hammond <jeff.science(a)gmail.com> >>> wrote: >>> >>>> As far as I know, Ethernet is not good at making asynchronous progress >>>> in hardware the way e.g. InfiniBand is. I would have thought that a >>>> dedicated progress thread would help, but it seems you tried that. Did you >>>> use your own progress thread or MPICH_ASYNC_PROGRESS=1? >>>> >>>> Jeff >>>> >>>> On Fri, Apr 3, 2015 at 10:10 AM, Lei Shi <lshi(a)ku.edu> wrote: >>>> >>>>> Huiwei, >>>>> >>>>> Thanks for your email. Your answer leads to my another question about >>>>> asynchronous MPI communication. >>>>> >>>>> I'm trying to do an overlapped communication/computing to speedup my >>>>> MPI code. I read some papers comparing some different approaches to do the >>>>> overlapped communication. The "naive" overlapped communication >>>>> implementation, which only use non-blocking mpi Isend/Irecv and the hybrid >>>>> approach using OpenMP and MPI together. In the hybrid approach, a separated >>>>> thread is use to do all non-blocking communications. Just exactly as you >>>>> said, the results indicate that current MPI implementations do not >>>>> support true asynchronous communication. >>>>> >>>>> If I use the naive approach, my code with non-blocking or blocking >>>>> send/recv gives me almost the same performance in term of Wtime. All >>>>> communications are postponed to MPI_Wait. >>>>> >>>>> I have tried calling mpi_test to push library to do communication >>>>> during iterations. And try to use a dedicated thread to do communication >>>>> and the other thread to do computing only. However, the performance gains >>>>> are very small or no gain at all. I'm wondering it is due to the hardware. >>>>> The cluster I tested uses 10G Ethernet card. >>>>> >>>>> >>>>> Best, >>>>> >>>>> Lei Shi >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Apr 3, 2015 at 8:49 AM, Huiwei Lu <huiweilu(a)mcs.anl.gov> >>>>> wrote: >>>>> >>>>>> Hi Lei, >>>>>> >>>>>> As far as I know, all current MPI implementations do not support true >>>>>> asynchronous communication for now. i.e., If there is no MPI calls in your >>>>>> iterations, MPICH will not be able to make progress on communication. >>>>>> >>>>>> One solution is to poll the MPI runtime regularly to make progress by >>>>>> inserting MPI_Test to your iteration (even though you do not want to check >>>>>> the data). >>>>>> >>>>>> Another solution is to enable MPI's asynchronous progress thread to >>>>>> make progress for you. >>>>>> >>>>>> -- >>>>>> Huiwei >>>>>> >>>>>> On Thu, Apr 2, 2015 at 11:44 PM, Lei Shi <lshi(a)ku.edu> wrote: >>>>>> >>>>>>> Hi Junchao, >>>>>>> >>>>>>> Thanks for your reply. For my case, I don't want to check the data >>>>>>> have been received or not. So I don't want to call MPI_Test or any function >>>>>>> to verify it. But my problem is like if I ignore calling the MPI_Wait, just >>>>>>> call Isend/Irev, my program freezes for several sec and then continues to >>>>>>> run. My guess is probably I messed up the MPI library internal buffer by >>>>>>> doing this. >>>>>>> >>>>>>> On Thu, Apr 2, 2015 at 7:25 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> >>>>>>> wrote: >>>>>>> >>>>>>>> Does MPI_Test fit your needs? >>>>>>>> >>>>>>>> --Junchao Zhang >>>>>>>> >>>>>>>> On Thu, Apr 2, 2015 at 7:16 PM, Lei Shi <lshi(a)ku.edu> wrote: >>>>>>>> >>>>>>>>> I want to use non-blocking send/rev MPI_Isend/MPI_Irev to do >>>>>>>>> communication. But in my case, I don't really care what kind of data I get >>>>>>>>> or it is ready to use or not. So I don't want to waste my time to do any >>>>>>>>> synchronization by calling MPI_Wait or etc API. >>>>>>>>> >>>>>>>>> But when I avoid calling MPI_Wait, my program is freezed several >>>>>>>>> secs after running some iterations (after multiple MPI_Isend/Irev >>>>>>>>> callings), then continues. It takes even more time than the case with >>>>>>>>> MPI_Wait. So my question is how to do a "true" non-blocking communication >>>>>>>>> without waiting for the data ready or not. Thanks. >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>>>> To manage subscription options or unsubscribe: >>>>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>>> To manage subscription options or unsubscribe: >>>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> jeff.science(a)gmail.com >>>> http://jeffhammond.github.io/ >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >>> >> > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to use non-blocking send/receive without calling MPI_Wait
by Lei Shi 07 Apr '15

07 Apr '15

Here is my pure MPI overlap version. I use intel traceanalyzer, the profiling shows that right now, communication only proceed when I call mpi_waitall on nodes with 10g network. /** pure mpi overlap **/ template<typename T> void CPR_NS_3D_Solver<T>::UpdateRes(T**q, T**res){ if(_n_proc>1) SendInterfaceSol(); //call isend/irecv to send msg 1 ResFromDivInvisFlux(q,res); //do local jobs if(_n_proc>1){ RevInterfaceSol(); //mpi_waitall for msg 1 if(vis_mode_) SendInterfaceCorrGrad(); //depends on msg 1 then snd msg 2 } if(vis_mode_) ResFromDivVisFlux(q,res); //computing, which depends on msg 1 if(_n_proc>1 && vis_mode_) RevInterfaceCorrGrad(); //mpi_waitall for msg 2 ResFromFluxCorrection(q,res); //computing, which depends on msg 1 and 2 } On Tue, Apr 7, 2015 at 2:39 AM, Lei Shi <lshi(a)ku.edu> wrote: > > > On Tue, Apr 7, 2015 at 2:37 AM, Lei Shi <leishi(a)ku.edu> wrote: > >> Hi Huiwei and Jeff, >> >> I use hybrid OpenMP/MPI to do overlap communication. So I put all >> communication in one dedicated OpenMP thread and computation in the other >> thread. For this case, I'm using intel MPI library. Probably I did some >> mistakes >> >> One version of my code using one dedicated thread to do messaging is like >> this >> >> /* hybrid mpi/openmp overlap **/template<typename T>void CPR_NS_3D_Solver<T>::UpdateRes(T**q, T**res){ >> int thread_id,n_thread; >> int sol_rev_flag=0,grad_rev_flag=0; >> >> // Explicitly disable dynamic teams >> omp_set_dynamic(0); >> // Use 2 threads for all consecutive parallel regions >> omp_set_num_threads(2); >> #pragma omp parallel default(shared) private(thread_id) >> { >> thread_id=omp_get_thread_num(); >> n_thread=omp_get_num_threads(); >> >> /** communication thread **/ >> if(thread_id==1){ >> SendInterfaceSol(); >> RevInterfaceSol();#pragma omp flush >> sol_rev_flag=1;#pragma omp flush(sol_rev_flag) >> } >> >> /** computation thread **/ >> if(thread_id==0){ >> ResFromDivInvisFlux(q,res); //local computation >> #pragma omp flush(sol_rev_flag) >> while(sol_rev_flag!=1){ #pragma omp flush(sol_rev_flag) >> }#pragma omp flush >> ResFromFluxCorrection(q,res); //depends on interface sol >> } >> }//end of omp >> } >> >> template<typename T> >> >> void CPR_NS_3D_Solver<T>::SendInterfaceSol(){ >> uint *n_if_to_proc=this->grid_->num_iface_proc; >> uint **if_to_proc=this->grid_->snd_iface_proc; >> uint **rev_if_to_f=this->grid_->rev_iface_proc; >> >> int tag=52; >> for(int p2=0;p2<_n_proc;++p2){ >> if(p2!=_proc_id){ >> int nif=n_if_to_proc[p2]; >> //pack data to send .... >> >> } >> } >> >> /** * Exchange interface sol **/ >> int n_proc_exchange=0; >> for(int z=0;z<_n_proc;++z){ >> int nif=n_if_to_proc[z]; >> >> //send data >> if(nif>0){ MPI_Isend(&snd_buf_[z][0],n_buf_[z],MPI_DOUBLE,z,tag, MPI_COMM_WORLD, &s_sol_req_[n_proc_exchange]); >> MPI_Irecv(&rev_buf_[z][0],n_buf_[z],MPI_DOUBLE,z,tag, MPI_COMM_WORLD, &r_sol_req_[n_proc_exchange]); >> n_proc_exchange++; >> } >> } >> >> } >> >> template<typename T> >> void CPR_NS_3D_Solver<T>::RevInterfaceSol(){ uint *n_if_to_proc=this->grid_->num_iface_proc; >> uint **if_to_proc=this->grid_->snd_iface_proc; >> uint **rev_if_to_f=this->grid_->rev_iface_proc; >> >> //wait >> if(n_proc_exchange_>0){ MPI_Waitall(n_proc_exchange_,s_sol_req_,MPI_STATUS_IGNORE); MPI_Waitall(n_proc_exchange_,r_sol_req_,MPI_STATUS_IGNORE); } >> >> /** store to local data structure **/ >> for(int z=0;z<_n_proc;++z){ >> int nif=n_if_to_proc[z]; >> >> if(nif>0){ >> >> //unpacking .... } >> } >> >> } >> >> >> >> >> >> >> Sincerely Yours, >> >> Lei Shi >> --------- >> >> On Fri, Apr 3, 2015 at 4:37 PM, Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> >>> As far as I know, Ethernet is not good at making asynchronous progress >>> in hardware the way e.g. InfiniBand is. I would have thought that a >>> dedicated progress thread would help, but it seems you tried that. Did you >>> use your own progress thread or MPICH_ASYNC_PROGRESS=1? >>> >>> Jeff >>> >>> On Fri, Apr 3, 2015 at 10:10 AM, Lei Shi <lshi(a)ku.edu> wrote: >>> >>>> Huiwei, >>>> >>>> Thanks for your email. Your answer leads to my another question about >>>> asynchronous MPI communication. >>>> >>>> I'm trying to do an overlapped communication/computing to speedup my >>>> MPI code. I read some papers comparing some different approaches to do the >>>> overlapped communication. The "naive" overlapped communication >>>> implementation, which only use non-blocking mpi Isend/Irecv and the hybrid >>>> approach using OpenMP and MPI together. In the hybrid approach, a separated >>>> thread is use to do all non-blocking communications. Just exactly as you >>>> said, the results indicate that current MPI implementations do not >>>> support true asynchronous communication. >>>> >>>> If I use the naive approach, my code with non-blocking or blocking >>>> send/recv gives me almost the same performance in term of Wtime. All >>>> communications are postponed to MPI_Wait. >>>> >>>> I have tried calling mpi_test to push library to do communication >>>> during iterations. And try to use a dedicated thread to do communication >>>> and the other thread to do computing only. However, the performance gains >>>> are very small or no gain at all. I'm wondering it is due to the hardware. >>>> The cluster I tested uses 10G Ethernet card. >>>> >>>> >>>> Best, >>>> >>>> Lei Shi >>>> >>>> >>>> >>>> >>>> >>>> On Fri, Apr 3, 2015 at 8:49 AM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: >>>> >>>>> Hi Lei, >>>>> >>>>> As far as I know, all current MPI implementations do not support true >>>>> asynchronous communication for now. i.e., If there is no MPI calls in your >>>>> iterations, MPICH will not be able to make progress on communication. >>>>> >>>>> One solution is to poll the MPI runtime regularly to make progress by >>>>> inserting MPI_Test to your iteration (even though you do not want to check >>>>> the data). >>>>> >>>>> Another solution is to enable MPI's asynchronous progress thread to >>>>> make progress for you. >>>>> >>>>> -- >>>>> Huiwei >>>>> >>>>> On Thu, Apr 2, 2015 at 11:44 PM, Lei Shi <lshi(a)ku.edu> wrote: >>>>> >>>>>> Hi Junchao, >>>>>> >>>>>> Thanks for your reply. For my case, I don't want to check the data >>>>>> have been received or not. So I don't want to call MPI_Test or any function >>>>>> to verify it. But my problem is like if I ignore calling the MPI_Wait, just >>>>>> call Isend/Irev, my program freezes for several sec and then continues to >>>>>> run. My guess is probably I messed up the MPI library internal buffer by >>>>>> doing this. >>>>>> >>>>>> On Thu, Apr 2, 2015 at 7:25 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> >>>>>> wrote: >>>>>> >>>>>>> Does MPI_Test fit your needs? >>>>>>> >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> On Thu, Apr 2, 2015 at 7:16 PM, Lei Shi <lshi(a)ku.edu> wrote: >>>>>>> >>>>>>>> I want to use non-blocking send/rev MPI_Isend/MPI_Irev to do >>>>>>>> communication. But in my case, I don't really care what kind of data I get >>>>>>>> or it is ready to use or not. So I don't want to waste my time to do any >>>>>>>> synchronization by calling MPI_Wait or etc API. >>>>>>>> >>>>>>>> But when I avoid calling MPI_Wait, my program is freezed several >>>>>>>> secs after running some iterations (after multiple MPI_Isend/Irev >>>>>>>> callings), then continues. It takes even more time than the case with >>>>>>>> MPI_Wait. So my question is how to do a "true" non-blocking communication >>>>>>>> without waiting for the data ready or not. Thanks. >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>>> To manage subscription options or unsubscribe: >>>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> http://jeffhammond.github.io/ >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to use non-blocking send/receive without calling MPI_Wait
by Lei Shi 07 Apr '15

07 Apr '15

On Tue, Apr 7, 2015 at 2:37 AM, Lei Shi <leishi(a)ku.edu> wrote: > Hi Huiwei and Jeff, > > I use hybrid OpenMP/MPI to do overlap communication. So I put all > communication in one dedicated OpenMP thread and computation in the other > thread. For this case, I'm using intel MPI library. Probably I did some > mistakes > > One version of my code using one dedicated thread to do messaging is like > this > > /* hybrid mpi/openmp overlap **/template<typename T>void CPR_NS_3D_Solver<T>::UpdateRes(T**q, T**res){ > int thread_id,n_thread; > int sol_rev_flag=0,grad_rev_flag=0; > > // Explicitly disable dynamic teams > omp_set_dynamic(0); > // Use 2 threads for all consecutive parallel regions > omp_set_num_threads(2); > #pragma omp parallel default(shared) private(thread_id) > { > thread_id=omp_get_thread_num(); > n_thread=omp_get_num_threads(); > > /** communication thread **/ > if(thread_id==1){ > SendInterfaceSol(); > RevInterfaceSol();#pragma omp flush > sol_rev_flag=1;#pragma omp flush(sol_rev_flag) > } > > /** computation thread **/ > if(thread_id==0){ > ResFromDivInvisFlux(q,res); //local computation > #pragma omp flush(sol_rev_flag) > while(sol_rev_flag!=1){ #pragma omp flush(sol_rev_flag) > }#pragma omp flush > ResFromFluxCorrection(q,res); //depends on interface sol > } > }//end of omp > } > > template<typename T> > > void CPR_NS_3D_Solver<T>::SendInterfaceSol(){ > uint *n_if_to_proc=this->grid_->num_iface_proc; > uint **if_to_proc=this->grid_->snd_iface_proc; > uint **rev_if_to_f=this->grid_->rev_iface_proc; > > int tag=52; > for(int p2=0;p2<_n_proc;++p2){ > if(p2!=_proc_id){ > int nif=n_if_to_proc[p2]; > //pack data to send .... > > } > } > > /** * Exchange interface sol **/ > int n_proc_exchange=0; > for(int z=0;z<_n_proc;++z){ > int nif=n_if_to_proc[z]; > > //send data > if(nif>0){ MPI_Isend(&snd_buf_[z][0],n_buf_[z],MPI_DOUBLE,z,tag, MPI_COMM_WORLD, &s_sol_req_[n_proc_exchange]); > MPI_Irecv(&rev_buf_[z][0],n_buf_[z],MPI_DOUBLE,z,tag, MPI_COMM_WORLD, &r_sol_req_[n_proc_exchange]); > n_proc_exchange++; > } > } > > } > > template<typename T> > void CPR_NS_3D_Solver<T>::RevInterfaceSol(){ uint *n_if_to_proc=this->grid_->num_iface_proc; > uint **if_to_proc=this->grid_->snd_iface_proc; > uint **rev_if_to_f=this->grid_->rev_iface_proc; > > //wait > if(n_proc_exchange_>0){ MPI_Waitall(n_proc_exchange_,s_sol_req_,MPI_STATUS_IGNORE); MPI_Waitall(n_proc_exchange_,r_sol_req_,MPI_STATUS_IGNORE); } > > /** store to local data structure **/ > for(int z=0;z<_n_proc;++z){ > int nif=n_if_to_proc[z]; > > if(nif>0){ > > //unpacking .... } > } > > } > > > > > > > Sincerely Yours, > > Lei Shi > --------- > > On Fri, Apr 3, 2015 at 4:37 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> As far as I know, Ethernet is not good at making asynchronous progress in >> hardware the way e.g. InfiniBand is. I would have thought that a dedicated >> progress thread would help, but it seems you tried that. Did you use your >> own progress thread or MPICH_ASYNC_PROGRESS=1? >> >> Jeff >> >> On Fri, Apr 3, 2015 at 10:10 AM, Lei Shi <lshi(a)ku.edu> wrote: >> >>> Huiwei, >>> >>> Thanks for your email. Your answer leads to my another question about >>> asynchronous MPI communication. >>> >>> I'm trying to do an overlapped communication/computing to speedup my MPI >>> code. I read some papers comparing some different approaches to do the >>> overlapped communication. The "naive" overlapped communication >>> implementation, which only use non-blocking mpi Isend/Irecv and the hybrid >>> approach using OpenMP and MPI together. In the hybrid approach, a separated >>> thread is use to do all non-blocking communications. Just exactly as you >>> said, the results indicate that current MPI implementations do not >>> support true asynchronous communication. >>> >>> If I use the naive approach, my code with non-blocking or blocking >>> send/recv gives me almost the same performance in term of Wtime. All >>> communications are postponed to MPI_Wait. >>> >>> I have tried calling mpi_test to push library to do communication during >>> iterations. And try to use a dedicated thread to do communication and the >>> other thread to do computing only. However, the performance gains are very >>> small or no gain at all. I'm wondering it is due to the hardware. The >>> cluster I tested uses 10G Ethernet card. >>> >>> >>> Best, >>> >>> Lei Shi >>> >>> >>> >>> >>> >>> On Fri, Apr 3, 2015 at 8:49 AM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: >>> >>>> Hi Lei, >>>> >>>> As far as I know, all current MPI implementations do not support true >>>> asynchronous communication for now. i.e., If there is no MPI calls in your >>>> iterations, MPICH will not be able to make progress on communication. >>>> >>>> One solution is to poll the MPI runtime regularly to make progress by >>>> inserting MPI_Test to your iteration (even though you do not want to check >>>> the data). >>>> >>>> Another solution is to enable MPI's asynchronous progress thread to >>>> make progress for you. >>>> >>>> -- >>>> Huiwei >>>> >>>> On Thu, Apr 2, 2015 at 11:44 PM, Lei Shi <lshi(a)ku.edu> wrote: >>>> >>>>> Hi Junchao, >>>>> >>>>> Thanks for your reply. For my case, I don't want to check the data >>>>> have been received or not. So I don't want to call MPI_Test or any function >>>>> to verify it. But my problem is like if I ignore calling the MPI_Wait, just >>>>> call Isend/Irev, my program freezes for several sec and then continues to >>>>> run. My guess is probably I messed up the MPI library internal buffer by >>>>> doing this. >>>>> >>>>> On Thu, Apr 2, 2015 at 7:25 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> >>>>> wrote: >>>>> >>>>>> Does MPI_Test fit your needs? >>>>>> >>>>>> --Junchao Zhang >>>>>> >>>>>> On Thu, Apr 2, 2015 at 7:16 PM, Lei Shi <lshi(a)ku.edu> wrote: >>>>>> >>>>>>> I want to use non-blocking send/rev MPI_Isend/MPI_Irev to do >>>>>>> communication. But in my case, I don't really care what kind of data I get >>>>>>> or it is ready to use or not. So I don't want to waste my time to do any >>>>>>> synchronization by calling MPI_Wait or etc API. >>>>>>> >>>>>>> But when I avoid calling MPI_Wait, my program is freezed several >>>>>>> secs after running some iterations (after multiple MPI_Isend/Irev >>>>>>> callings), then continues. It takes even more time than the case with >>>>>>> MPI_Wait. So my question is how to do a "true" non-blocking communication >>>>>>> without waiting for the data ready or not. Thanks. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [OMPI users] Buffered sends are evil?
by Jeff Hammond 06 Apr '15

06 Apr '15

While we are tilting at windmills, can we also discuss the evils of MPI_Cancel for MPI_Send, everything about MPI_Alltoallw, how MPI_Reduce_scatter is named wrong, and any number of other pet peeves that people have about MPI-3? :-D The MPI standard contains many useful functions and at least a handful of stupid ones. This is remarkably similar to other outputs of the design-by-committee process and can be observed in OpenMP 4.0, C++14, Fortran 2008, and probably every other standardized parallel programming interface in use today. Fortunately, judicious parallel programmers know that less is more and generally focus on using the useful functions effectively, while ignoring the less useful ones, and it's usually not hard to tell the difference. Jeff PS I used MPI_Bsend once and found it superior to the alternative of MPI_Isend+MPI_Request_free for sending fire-and-forget acks, because it forced the implementation to do what I wanted and the performance improved as a result. On Fri, Apr 3, 2015 at 8:34 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > Fair enough. > > My main point should probably be summarized as: MPI_BSEND isn't worth it; there are other, less-confusing, generally-more-optimized alternatives. > > > > >> On Apr 3, 2015, at 11:20 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >> Jeff, >> >> Your blog post seems to confuse what implementations currently do with what Bsend is capable of. If users really wanted it (e.g., a big customer asked for it), every implementation will optimize the crap out of it. The problem is that every few users really care for it, so there's not been a good incentive for implementations to optimize it. >> >> Coming to the technical aspects, bsend doesn't require copying into the user buffer, if you have internal buffer resources. It only guarantees that Bsend will not block if enough user buffer space is available. If you are blocking for progress anyway, I'm not sure the extra copy would matter too much -- it matters some, of course, but likely not to the extent of a full copy cost. Also, the semantics it provides are different -- guaranteed nonblocking nature when there's buffer space available. It's like saying Ssend is not as efficient as send. That's true, but those are different semantics. >> >> Having said that, I do agree with some of the shortcomings you pointed out -- specifically, you can only attach one buffer. I'd add to the list with one more shortcoming: It's not communicator safe. That is, if I attach a buffer, some other library I linked with might chew up my buffer space. So the nonblocking guarantee is kind of bogus at that point. >> >> -- Pavan >> >>> On Apr 3, 2015, at 5:30 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: >>> >>> Yes. I think the blog post gives 10 excellent reasons why. :-) >>> >>> >>>> On Apr 3, 2015, at 2:40 AM, Lei Shi <leishi(a)ku.edu> wrote: >>>> >>>> Hello, >>>> >>>> I want to use buffered sends. Read a blog said it is evil, http://blogs.cisco.com/performance/top-10-reasons-why-buffered-sends-are-ev… >>>> >>>> Is it true or not? Thanks! >>>> >>>> Sincerely Yours, >>>> >>>> Lei Shi >>>> --------- >>>> >>>> _______________________________________________ >>>> users mailing list >>>> users(a)open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: http://www.open-mpi.org/community/lists/users/2015/04/26597.php >>> >>> >>> -- >>> Jeff Squyres >>> jsquyres(a)cisco.com >>> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> -- >> Pavan Balaji ✉️ >> http://www.mcs.anl.gov/~balaji >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] unsubscribe
by Jeff Hammond 06 Apr '15

06 Apr '15

Note also that https://lists.mpich.org/mailman/listinfo/discuss has all the powers of a sentient being, provided the operator knows how to use it :-) Jeff On Mon, Apr 6, 2015 at 7:44 AM, "Antonio J. Peña" <apenya(a)mcs.anl.gov> wrote: > > Hi Thierry, > > The e-mail address you're writing from is not currently subscribed to our > mailing list. May you have another one forwarded or an e-mail alias? > > Best, > Antonio > > > On 04/06/2015 01:55 AM, Carron, Thierry wrote: > > > > > > Thierry Carron > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Antonio J. Peña > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847 > apenya(a)mcs.anl.gov > www.mcs.anl.gov/~apenya > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] unsubscribe
by "Antonio J. Peña" 06 Apr '15

06 Apr '15

Hi Thierry, The e-mail address you're writing from is not currently subscribed to our mailing list. May you have another one forwarded or an e-mail alias? Best, Antonio On 04/06/2015 01:55 AM, Carron, Thierry wrote: > > Thierry Carron > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

unsubscribe
by Carron, Thierry 06 Apr '15

06 Apr '15

Thierry Carron _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to use non-blocking send/receive without calling MPI_Wait
by Lei Shi 04 Apr '15

04 Apr '15

Yes, I tested it and solved my issue. Thanks again!! On Sat, Apr 4, 2015 at 7:01 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > MPI_REQUEST_FREE. Please read the remaining emails on this thread. > > -- Pavan > > > On Apr 3, 2015, at 7:56 PM, Lei Shi <lshi(a)ku.edu> wrote: > > > > Sorry. I just take a quick look of that slides this afternoon. Got the > details now. > > > > So can I make a quick conclusion that there is no way to avoid MPI_Wait > or MPI_test with non-blocking send/receive. Even I don't care I have > received the message or not. > > > > > > On Fri, Apr 3, 2015 at 5:23 PM, Jeff Squyres (jsquyres) < > jsquyres(a)cisco.com> wrote: > > On Apr 3, 2015, at 6:09 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > > > > > I didn't look through the video, but what you are saying is not true. > You can free the request in the application (this simply reduces the > reference count to that request). MPI will still maintain one reference > count to the request and will only free it when the data transfer completes. > > > > That's correct. The point of that blog entry is that you have to tell > MPI that you're done with a request somehow, otherwise MPI *has* to keep > those resources allocated. > > > > MPI_TEST*/MPI_WAIT* are the usual ways to do this. MPI_REQUEST_FREE is > also a valid method, but, as has been stated, isn't advisable. > > > > -- > > Jeff Squyres > > jsquyres(a)cisco.com > > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to use non-blocking send/receive without calling MPI_Wait
by Balaji, Pavan 04 Apr '15

04 Apr '15

MPI_REQUEST_FREE. Please read the remaining emails on this thread. -- Pavan > On Apr 3, 2015, at 7:56 PM, Lei Shi <lshi(a)ku.edu> wrote: > > Sorry. I just take a quick look of that slides this afternoon. Got the details now. > > So can I make a quick conclusion that there is no way to avoid MPI_Wait or MPI_test with non-blocking send/receive. Even I don't care I have received the message or not. > > > On Fri, Apr 3, 2015 at 5:23 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > On Apr 3, 2015, at 6:09 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > > > I didn't look through the video, but what you are saying is not true. You can free the request in the application (this simply reduces the reference count to that request). MPI will still maintain one reference count to the request and will only free it when the data transfer completes. > > That's correct. The point of that blog entry is that you have to tell MPI that you're done with a request somehow, otherwise MPI *has* to keep those resources allocated. > > MPI_TEST*/MPI_WAIT* are the usual ways to do this. MPI_REQUEST_FREE is also a valid method, but, as has been stated, isn't advisable. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0