----- Original Message -----

From: "Xin Zhao" <[email protected]> To: "Hal Finkel" <[email protected]> Cc: "Pavan Balaji" <[email protected]>, [email protected] Sent: Monday, June 30, 2014 3:59:50 PM Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED

Hi Hal,

The functions of poking progress engine are MPID_Progress_poke(), MPID_Progress_wait() and MPID_Progress_test(), but they are not provided with a request pointer directly. By saying "pass the single request pointer", I mean we pass it when issuing operations (location I mentioned in the last email), and if it is not completed, the request will be enqueued in a queue in the lower layer. When we call MPID_Progress_ functions, the progress engine will process on that queue and try to make more requests completed.

Thanks, now I understand what you meant. -Hal

Xin ________________________________________ From: Hal Finkel [[email protected]] Sent: Monday, June 30, 2014 3:45 PM To: Zhao, Xin Cc: Pavan Balaji; [email protected] Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED

----- Original Message -----

...

From: "Xin Zhao" <[email protected]> To: "Hal Finkel" <[email protected]> Cc: "Pavan Balaji" <[email protected]>, [email protected] Sent: Monday, June 30, 2014 3:39:20 PM Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED

Hi Hal,

I am not sure if I understand you correctly. If you mean the function that CH3 uses to pass the message to the lower layer, it indeed tests if it is completed or not upon return from nemesis. If completed, CH3 will free that operation immediately, if not, it will leave that operation in the list. You can refer to L1087~L1097 in src/mpid/ch3/src/ch3u_rma_sync.c in latest mpich/master about this.

No, I mean where is the code in the CH3 layer that calls the nemesis layer (pokes the progress engine?) providing it only with the single request pointer? In short, where is this "We didn't pass the list to the nemesis layer but only pass the single request pointer"?

Thanks again, Hal

...

Xin ________________________________________ From: Hal Finkel [[email protected]] Sent: Monday, June 30, 2014 3:07 PM To: Zhao, Xin Cc: Pavan Balaji; [email protected] Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED

----- Original Message -----

...

From: "Xin Zhao" <[email protected]> To: "Hal Finkel" <[email protected]>, [email protected] Cc: "Pavan Balaji" <[email protected]> Sent: Monday, June 30, 2014 2:14:15 PM Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED

Hi Hal,

Currently the RMA operation list is implemented in CH3 layer, whereas the progress engine is implemented in nemesis layer. We didn't pass the list to the nemesis layer but only pass the single request pointer, so the progress engine now only modify on the request pointer.

Thanks for the explanation. FWIW, I don't understand why the calling code (in CH3) could not test for complete upon return from nemesis and handle it then. Which function handles this?

-Hal

...

This is one of the problems we have with the current separation of the CH3 and nemesis layers, we plan to merge these layers in future and this should be fixed at that time.

Thanks, Xin ________________________________________ From: Hal Finkel [[email protected]] Sent: Monday, June 30, 2014 1:28 PM To: [email protected] Cc: Zhao, Xin; Pavan Balaji Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED

Xin, et al.,

Regarding 283319f5d9d1cdbcf016dc0ff8ab284114c0be7e (which added MPIR_CVAR_CH3_RMA_GC_NUM_TESTED and associated code): If a request is completed by the progress engine, why can't the progress engine move it to the beginning of the list (or the end), and have rma_list_gc only search from the beginning (or end) until it finds an incomplete request?

Thanks in advance, Hal

-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory