While on the same topic, I am aware of the Paraver tool-kit and the MPItrace utiliy to graphically represent the output that is being generated. But, I have been waiting for almost three days now, to gain access to the Paraver source code. It would be of great help if someone can suggest an alternative to Paraver. Thanks, Krishna Chaitanya K On Fri, Mar 14, 2008 at 11:09 AM, Krishna Chaitanya <[email protected]> wrote:

Hi, I was playing with the Open MPI code last fall and I found many references to PERUSE. On searching further, I realised its role and found out that it has not yet been developed for MPICH. I have been working on the sock device with the blocking and non-blocking modes for point-to-point calls. The prototype currently implements all the events specified by the PERUSE specification, excluding : PERUSE_COMM_REQ_XFER_BEGIN, and PERUSE_COMM_REQ_XFER_END. I am still working on these, as the same function, MPIDU_Socki_handle_write, is used to transfer both the control information and the data. With respect to the API, the functions that are yet to be implemented are PERUSE_Event_propogate, PERUSE_Query_envirnoment and PERUSE_Query_queue_event_scope. These functions are not critical with regard to functioning of the prototype. As a sample program, I tried the following, Please find the corresponding output files for rank0 and rank1 attached. They indicate the events that are occuring and the time of occurence.

( rank 0 ) (rank 1)

SENDER RECEIVER MPI_Init() MPI_Init() PERUSE_Init() PERUSE_Init() /* .......... /* .......... peruse configuration peruse configuration ............. */ ............. */ buf1 = Message 1; buf2 = Message 2; buf3 = Message 3;

MPI_Isend(buf1 , tag =1, ...) MPI_Irecv(.... , tag =3, ...) MPI_Isend(buf2 , tag =2, ...) MPI_Irecv(.... , tag =2, ...) MPI_Isend( buf3 , tag =3, ...) MPI_Irecv(.... , tag =1, ...) MPI_Wait() MPI_Wait() MPI_Wait() MPI_Wait() MPI_Wait() MPI_Wait()

PERUSE_Finalize() PERUSE_Finalize() MPI_Finalize() MPI_Finalize()

Please do let me know if you happen to find anything possibly wrong.

Thanks, Krishna Chaitanya K, Final Year B-TECH, Dept. of Information Technology, National Institute of Technology,Karnataka, India

On Thu, Mar 13, 2008 at 4:31 PM, Rajeev Thakur <[email protected]> wrote:

...

Hi Krishna, That is good to know. How did you even know about PERUSE? (I didn't think it was that well known.) How much of the interface have you implemented? We are interested in knowing more about what you have done.

Regards, Rajeev

------------------------------ *From:* Krishna Chaitanya [mailto:[email protected]] *Sent:* Thursday, March 13, 2008 7:51 AM *To:* [email protected] *Cc:* [email protected]; [email protected] *Subject:* Development of the PERUSE interface for MPICH2

Hi, As my final year B-Tech project, I have been working on developing the PERUSE <http://www.mpi-peruse.org/> interface for MPICH2. PERUSE is an attempt to expose fine-grained information about the internals of an MPI implementation by monitoring certain events of interest and the time elapsed between such events. The interface allows the user to define a set of events and register callbacks, which are to be invoked by the MPI library as and when an event of interest occurs. PERUSE has been successfully integrated with Open-MPI. I have the prototype working for blocking and non_blocking mode of data transfer with the sock device. I have experimented with a few examples having multiple senders and a slow receiver and have been able to generate meaningful output. Since the data generated is quite voluminous, I am now trying to present it in a graphical,user friendly manner. I wish to have your feedback on this project to look at the possible areas of improvement and further work. Please let me know if you are interested.

Thanks, Krishna Chaitanya K, Final Year B-Tech, Dept. of Information Technology, National Institute of Technology,Karnataka, India.

-- In the middle of difficulty, lies opportunity

-- In the middle of difficulty, lies opportunity

-- In the middle of difficulty, lies opportunity

We've looked at PERUSE (and we were involved in the early discussions on PERUSE) but for many of our platforms, PERUSE has both too high an overhead and does not match the underlying structure of the communication system (it isn't bad if the system is ch3:sock, but its not a good match to dcmf or to speculative send approaches used in some other MPICH devices). Instead, we proposed some time ago a lighter-weight interface that was also more flexible, allowing the MPI implementation to define instrumentation events that the implementation considered significant and low cost. Rather than a call-back mechanism (which isn't bad when you are talking about commodity network interfaces on 3 GHz processors but is a potential problem on ultrafast, memory-oriented interfaces on massive numbers of lower frequency processors), we've considered a lightweight, simple event log mechanism instead. What we're most interested in is an abstract instrumentation interface with events of the developer's choosing. Currently, we're developing a document that describes our light-weight interface; I can send you a copy when its ready. Bill On Mar 14, 2008, at 10:09 AM, Krishna Chaitanya wrote:

Hi, I was playing with the Open MPI code last fall and I found many references to PERUSE. On searching further, I realised its role and found out that it has not yet been developed for MPICH. I have been working on the sock device with the blocking and non-blocking modes for point-to-point calls. The prototype currently implements all the events specified by the PERUSE specification, excluding : PERUSE_COMM_REQ_XFER_BEGIN, and PERUSE_COMM_REQ_XFER_END. I am still working on these, as the same function, MPIDU_Socki_handle_write, is used to transfer both the control information and the data. With respect to the API, the functions that are yet to be implemented are PERUSE_Event_propogate, PERUSE_Query_envirnoment and PERUSE_Query_queue_event_scope. These functions are not critical with regard to functioning of the prototype. As a sample program, I tried the following, Please find the corresponding output files for rank0 and rank1 attached. They indicate the events that are occuring and the time of occurence.

( rank 0 ) (rank 1)

SENDER RECEIVER MPI_Init() MPI_Init() PERUSE_Init() PERUSE_Init() /* .......... / * .......... peruse configuration peruse configuration ............. */ ............. */ buf1 = Message 1; buf2 = Message 2; buf3 = Message 3;

MPI_Isend(buf1 , tag =1, ...) MPI_Irecv(.... , tag =3, ...) MPI_Isend(buf2 , tag =2, ...) MPI_Irecv(.... , tag =2, ...) MPI_Isend( buf3 , tag =3, ...) MPI_Irecv (.... , tag =1, ...) MPI_Wait() MPI_Wait() MPI_Wait() MPI_Wait() MPI_Wait() MPI_Wait()

PERUSE_Finalize() PERUSE_Finalize() MPI_Finalize() MPI_Finalize()

Please do let me know if you happen to find anything possibly wrong.

Thanks, Krishna Chaitanya K, Final Year B-TECH, Dept. of Information Technology, National Institute of Technology,Karnataka, India

On Thu, Mar 13, 2008 at 4:31 PM, Rajeev Thakur <[email protected]> wrote: Hi Krishna, That is good to know. How did you even know about PERUSE? (I didn't think it was that well known.) How much of the interface have you implemented? We are interested in knowing more about what you have done.

Regards, Rajeev

From: Krishna Chaitanya [mailto:[email protected]] Sent: Thursday, March 13, 2008 7:51 AM To: [email protected] Cc: [email protected]; [email protected] Subject: Development of the PERUSE interface for MPICH2

Hi, As my final year B-Tech project, I have been working on developing the PERUSE interface for MPICH2. PERUSE is an attempt to expose fine-grained information about the internals of an MPI implementation by monitoring certain events of interest and the time elapsed between such events. The interface allows the user to define a set of events and register callbacks, which are to be invoked by the MPI library as and when an event of interest occurs. PERUSE has been successfully integrated with Open-MPI. I have the prototype working for blocking and non_blocking mode of data transfer with the sock device. I have experimented with a few examples having multiple senders and a slow receiver and have been able to generate meaningful output. Since the data generated is quite voluminous, I am now trying to present it in a graphical,user friendly manner. I wish to have your feedback on this project to look at the possible areas of improvement and further work. Please let me know if you are interested.

Thanks, Krishna Chaitanya K, Final Year B-Tech, Dept. of Information Technology, National Institute of Technology,Karnataka, India.

-- In the middle of difficulty, lies opportunity

-- In the middle of difficulty, lies opportunity<rank0><rank1>

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

Like all of MPICH2, this will be open source. The specification will be open for comment and revision as well. Bill On Mar 20, 2008, at 8:37 AM, Krishna Chaitanya wrote:

Dear Sir, Thank you for the feedback.

...

Currently, we're developing a document that describes our light- weight interface; I can send you a copy when its ready. I appreciate that. Would this be an open source project or will the development be limited to a specific team?

Regards, Krishna Chaitanya K

On Wed, Mar 19, 2008 at 10:10 AM, William Gropp <[email protected]> wrote: We've looked at PERUSE (and we were involved in the early discussions on PERUSE) but for many of our platforms, PERUSE has both too high an overhead and does not match the underlying structure of the communication system (it isn't bad if the system is ch3:sock, but its not a good match to dcmf or to speculative send approaches used in some other MPICH devices). Instead, we proposed some time ago a lighter-weight interface that was also more flexible, allowing the MPI implementation to define instrumentation events that the implementation considered significant and low cost. Rather than a call-back mechanism (which isn't bad when you are talking about commodity network interfaces on 3 GHz processors but is a potential problem on ultrafast, memory- oriented interfaces on massive numbers of lower frequency processors), we've considered a lightweight, simple event log mechanism instead.

What we're most interested in is an abstract instrumentation interface with events of the developer's choosing. Currently, we're developing a document that describes our light-weight interface; I can send you a copy when its ready.

Bill

On Mar 14, 2008, at 10:09 AM, Krishna Chaitanya wrote:

...

Hi, I was playing with the Open MPI code last fall and I found many references to PERUSE. On searching further, I realised its role and found out that it has not yet been developed for MPICH. I have been working on the sock device with the blocking and non-blocking modes for point-to-point calls. The prototype currently implements all the events specified by the PERUSE specification, excluding : PERUSE_COMM_REQ_XFER_BEGIN, and PERUSE_COMM_REQ_XFER_END. I am still working on these, as the same function, MPIDU_Socki_handle_write, is used to transfer both the control information and the data. With respect to the API, the functions that are yet to be implemented are PERUSE_Event_propogate, PERUSE_Query_envirnoment and PERUSE_Query_queue_event_scope. These functions are not critical with regard to functioning of the prototype. As a sample program, I tried the following, Please find the corresponding output files for rank0 and rank1 attached. They indicate the events that are occuring and the time of occurence.

( rank 0 ) (rank 1)

SENDER RECEIVER MPI_Init() MPI_Init() PERUSE_Init() PERUSE_Init() /* .......... / * .......... peruse configuration peruse configuration ............. */ ............. */ buf1 = Message 1; buf2 = Message 2; buf3 = Message 3;

MPI_Isend(buf1 , tag =1, ...) MPI_Irecv(.... , tag =3, ...) MPI_Isend(buf2 , tag =2, ...) MPI_Irecv (.... , tag =2, ...) MPI_Isend( buf3 , tag =3, ...) MPI_Irecv (.... , tag =1, ...) MPI_Wait() MPI_Wait() MPI_Wait() MPI_Wait() MPI_Wait() MPI_Wait()

PERUSE_Finalize() PERUSE_Finalize() MPI_Finalize() MPI_Finalize()

Please do let me know if you happen to find anything possibly wrong.

Thanks, Krishna Chaitanya K, Final Year B-TECH, Dept. of Information Technology, National Institute of Technology,Karnataka, India

On Thu, Mar 13, 2008 at 4:31 PM, Rajeev Thakur <[email protected]> wrote: Hi Krishna, That is good to know. How did you even know about PERUSE? (I didn't think it was that well known.) How much of the interface have you implemented? We are interested in knowing more about what you have done.

Regards, Rajeev

From: Krishna Chaitanya [mailto:[email protected]] Sent: Thursday, March 13, 2008 7:51 AM To: [email protected] Cc: [email protected]; [email protected] Subject: Development of the PERUSE interface for MPICH2

Hi, As my final year B-Tech project, I have been working on developing the PERUSE interface for MPICH2. PERUSE is an attempt to expose fine-grained information about the internals of an MPI implementation by monitoring certain events of interest and the time elapsed between such events. The interface allows the user to define a set of events and register callbacks, which are to be invoked by the MPI library as and when an event of interest occurs. PERUSE has been successfully integrated with Open-MPI. I have the prototype working for blocking and non_blocking mode of data transfer with the sock device. I have experimented with a few examples having multiple senders and a slow receiver and have been able to generate meaningful output. Since the data generated is quite voluminous, I am now trying to present it in a graphical,user friendly manner. I wish to have your feedback on this project to look at the possible areas of improvement and further work. Please let me know if you are interested.

Thanks, Krishna Chaitanya K, Final Year B-Tech, Dept. of Information Technology, National Institute of Technology,Karnataka, India.

-- In the middle of difficulty, lies opportunity

-- In the middle of difficulty, lies opportunity<rank0><rank1>

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- In the middle of difficulty, lies opportunity

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign