Subject: RE: Case: CAS-17789-C4N5V0 Exposing MPI Objects for Debugging tool in Totalview?
From: "Mohror, Kathryn" <[email-masked]>
To: "Schilling, Herbert W. (GRC-VE00)" <[email-masked]>
Cc: John DelSignore <[email-masked]>, Rogue Wave Support <[email-masked]>, Jeff Squyres <[email-masked]>, "Castain, Ralph H" <[email-masked]>, "Robinson, Calvin R. (GRC-VE00)" <[email-masked]>
Date: Thu, 17 Dec 2015 15:44:13 +0000

Hi Herb,

This is great – thank you!

Talk with you soon.

Kathryn

_________________________________________________________________
Kathryn Mohror, [email-masked]<mailto:[email-masked]>, http://scalability.llnl.gov/
Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA

From: Schilling, Herbert W. (GRC-VE00) [mailto:[email-masked]]
Sent: Thursday, December 17, 2015 7:43 AM
To: <[email-masked]>
Cc: John DelSignore; Rogue Wave Support; Jeff Squyres; Castain, Ralph H; Robinson, Calvin R. (GRC-VE00)
Subject: Re: Case: CAS-17789-C4N5V0  Exposing MPI Objects for Debugging tool in Totalview? 

Hi Kathryn,

 I will be able to attend but Calvin will not be able to. We came up with a list that we can talk about

  1.  State of the message queues

  1.  Contents of opaque MPI objects

     *   Like those created by MPI_Create_type_struct

     *   View file set views and memory mappings

  1.  When using multiple communicators, knowing getting at info about how processes are paired with ranks in communicators is useful

  1.  Find out about message queue sizes

  1.  Maybe a “lint” like call that checks the state of MPI to see if some MPI data structures got stomped on

  1.  Get info about communicators

I should warn you that while we have done work with MPI, I would definitely not consider us to be experts. Mostly we have been trying to help a researcher deal with bugs and performance issues with his large MPI code. But in the near future we will be helping other projects on parallelizing their codes.

See in a bit !

Thanks again for this opportunity.

On Dec 11, 2015, at 3:38 PM, Kathryn Mohror <[email-masked]<mailto:[email-masked]>> wrote:

Hi Herb,

It looks like the folks that would benefit most from your input will be able to attend the Dec 17 meeting. So, the meeting is a go!

It will help us during the meeting if you have some sort of “wish list” of the MPI objects and information about them that you think you would find most useful. Then, we can discuss each of your wishes to decide if they are feasible, and perhaps if we can provide more or something slightly different that would also be helpful.

The information for joining the meeting is here:
https://github.com/mpiwg-tools/tools-issues/wiki/Meetings

We use Cisco’s Webex for meetings so we can share documents on the screen. If you have not used it before, it is a free client download.

Thanks! And let me know if you have any questions.

Kathryn

_________________________________________________________________
Kathryn Mohror, [email-masked]<mailto:[email-masked]>, http://scalability.llnl.gov/
Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA

From: Schilling, Herbert W. (GRC-VE00) [mailto:[email-masked]<http://nasa.gov/>]
Sent: Tuesday, December 8, 2015 5:37 PM
To: Mohror, Kathryn
Cc: John DelSignore; Rogue Wave Support; Jeff Squyres; Kathryn Mohror; Castain, Ralph H; Robinson, Calvin R. (GRC-VE00)
Subject: Re: Case: CAS-17789-C4N5V0  Exposing MPI Objects for Debugging tool in Totalview? 

Hi Kathryn,

Sure, I could participate in one meeting and right now I can attend the Dec 17 meeting.

Thanks for the offer!

On Dec 8, 2015, at 4:46 PM, "Mohror, Kathryn" <[email-masked]<mailto:[email-masked]>>
 wrote:

Hi Herb,

As far as your participation in the MPI Tools Working Group goes, if you don’t have much time, perhaps you could participate in one meeting (or a couple meetings) to talk about what MPI objects and information would be useful to you?

If that sounds like what you want to do, perhaps we could use a portion of our next meeting to discuss this. I am guessing that since John is working on this now, knowing what you are interested will be more valuable to him sooner rather than later. John, let me know if I am wrong though, and we can push this off until after the holidays.

Our next meeting is scheduled for Dec 17 at 8:00 AM PST. If that time is not good, we can try for January some time.

Thanks,
Kathryn

_________________________________________________________________
Kathryn Mohror, [email-masked]<mailto:[email-masked]>, http://scalability.llnl.gov/
Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA

From: Schilling, Herbert W. (GRC-VE00) [mailto:[email-masked]<http://nasa.gov/>]
Sent: Monday, December 7, 2015 2:36 PM
To: John DelSignore
Cc: Rogue Wave Support; Jeff Squyres; Kathryn Mohror; Castain, Ralph H; Robinson, Calvin R. (GRC-VE00)
Subject: Re: Case: CAS-17789-C4N5V0  Exposing MPI Objects for Debugging tool in Totalview? 

Hi John,

 Yes, I am just interested in it as a debugger user. I would be happy to participate as a user to give you what we are interested in. I cc'ed Calvin Robinson on this email. He works with me at NASA and we are both interested in this.

 If I need to be part of the MPI Tools Group to make that happen it is OK with me. But, to be honest, I don't have the time or skill level to contribute much else to the Group. I definitely would love to help with the creation of MPI debugging tools.

 Thanks

On Dec 7, 2015, at 12:57 PM, John DelSignore <[email-masked]<mailto:[email-masked]>>
 wrote:

Hi Herb,

I'm not sure what your interest is in the MPI Handles stuff, but I'm assuming that it is as a debugger user. If so, I'm wondering if you'd be willing to participate "as a user" to tell us tools guys what MPI Handles information you'd like to see and how you'd like to see it presented. Nothing too onerous, but I've always found it much more productive to ask the users what features they want, than have us tools guys guess, and get it wrong. :-)

Any interest in joining the MPI Tools Group (assuming it's OK with Kathryn), or at least give us feedback on the user-visible debugging features that could be enabled by the MPI Handles work?

Cheers, John D.

Schilling, Herbert W. (GRC-VE00) wrote:
Hi John,

 Thanks for the great information on this development. It will be a great new capability to have this.

On Dec 7, 2015, at 12:20 PM, John DelSignore <[email-masked]<mailto:[email-masked]>>
 wrote:

Hi,

(Adding folks from the MPI Tools Group and Open MPI to the CC list.)

Interesting that this topic came up now!

Why? Because it just so happens that I'm currently spending a portion of my "vacation" time working on the MPI Handles Introspection Interface, which I consider "volunteer" work for the MPI Tools Group. I can confirm that currently neither TotalView nor Open MPI contains support for the MPI Handles Introspection Interface, but we are working towards making that happen. Peter, for your reference, support for MPI Handles is logged in our issue data base TVT-19578.

Here's the status...

Laust Brock-Nannestad, the primary author of the paper, did some prototyping work on TotalView as a 2014 summer student working at LLNL. I have recently merged his changes into the current development head of TotalView, and I'm working on cleaning up his code. I have the basics of hist ".mpidbg" working (I decided to rename the TV CLI command to "dmpi") using an old development version of Open MPI (circa v1.8 to v1.9); that version has a working MPIR Process Acquisition Interface and a working MPI Handles interface, but the MQD (Message Queue Dumping) interface is broken in that version. I think that version of Open MPI is "good enough" for me to implement the support described in the paper, but it's not ready for production use.

Jeff Squyres is pushing forward the implementation on the Open MPI side, and we recently agreed that it makes sense to use the Open MPI development master moving forward, which is Open MPI v2.0.0. Unfortunately, I discovered that there are a few things broken in the current version (like basic debugging support for the MPIR  and the MPI Handles Introspection Interface itself), but Jeff and Ralph Castain are working on fixing them.

Note that the current MPI Handles specification is a work in progress and subject to change as Jeff and I nudge things along. Hopefully, the specification, and the TotalView and Open MPI implementations will all come together at some point, at which point the MPI Tools Group will work on getting it standardized and implemented in other MPIs.

Cheers, John D.

Rogue Wave Support wrote:
Hello Herb,

I can tell you that it is not in the production version of TotalView, as I just tried .mpidbg inside the latest version and it wasn't recognized.  Of course there is always the possibility that the command name was changed, and that's why I'm not finding it, but I haven't really heard anything about this functionality, so I'm guessing it's still not mainstream.   I'm copying John Delsignore, as he's one of the authors and probably has a better idea of the state of things.  His is currently using his vacation time up before it disappears at the end of the year, but I'm pretty sure he keeps an eye open for interesting bits to comment on, so he may well pipe in.  Since this requires changes to the MPI interface itself, it may be that hasn't taken place yet.    I've got a copy of Open MPI 1.10, and I'm not seeing any references to this functionality, though again, name changes may be hiding it.

So, at this point, it looks like we haven't really integrated it, but maybe we have a version you could test with, assuming I can lay my hands on it.

Regards,
Peter Thompson

Software Support
Rogue Wave Software, Inc.
Accelerating Great Code
Contact Support |www.roguewave.com/support/contact-support.aspx<http://www.roguewave.com/support/contact-support.aspx>

------------------- Original Message -------------------
From: Herb Schilling
Received:
To: Rogue Wave Support; Rogue Wave Support; Rogue Wave Support
Subject: Exposing MPI Objects for Debugging tool in Totalview?

Hi Peter,

 Was this ever implemented in Totalview?

http://sc14.supercomputing.org/sites/all/themes/sc14/files/archive/tech_poster/poster_files/post284s2-file3.pdf

Looks very useful!

Thanks

---------
Herb Schilling
Scientific Applications and Visualization Team
http://ocio.grc.nasa.gov/gvis
Office of the Chief Information Officer
NASA Glenn Research Center
[email-masked]<mailto:[email-masked]>

---------
Herb Schilling
Scientific Applications and Visualization Team
http://ocio.grc.nasa.gov/gvis
Office of the Chief Information Officer
NASA Glenn Research Center
[email-masked]<mailto:[email-masked]>

---------
Herb Schilling
Scientific Applications and Visualization Team
http://ocio.grc.nasa.gov/gvis
Office of the Chief Information Officer
NASA Glenn Research Center
[email-masked]<mailto:[email-masked]>

---------
Herb Schilling
Scientific Applications and Visualization Team
http://ocio.grc.nasa.gov/gvis
Office of the Chief Information Officer
NASA Glenn Research Center
[email-masked]<mailto:[email-masked]>

---------
Herb Schilling
Scientific Applications and Visualization Team
http://ocio.grc.nasa.gov/gvis
Office of the Chief Information Officer
NASA Glenn Research Center
[email-masked]<mailto:[email-masked]>