I think I need to override MPIU_Malloc myself for this but I will also try --enable-g=mem and MPIU_trspace to see how the information I get from that lines up with what I get from the BGQ SPI (which can track everything: heap, stack, shm, ...). Thanks for all the pointers. Jeff On Sat, Feb 23, 2013 at 10:16 AM, Dave Goodell <[email protected]> wrote:
There are two kinds of memory that MPICH with ch3:nemesis allocates:
1) regular heap memory via malloc (with interposition possible via MPIU_Malloc and friends)
2) shared memory via one of the usual mechanisms (mmap, shm_open, etc.)
You can determine the current amount of #1 by calling "MPIU_trspace" as long as you configure with "--enable-g=mem" (or some superset thereof). You'll get some additional performance overhead when allocating/freeing memory if you configure this way, since MPICH is doing some additional allocation and bookkeeping in order to track this information for you. If you need a lower overhead method for doing this, you'll need to modify/intercept MPIU_Malloc and friends yourself. You'll need an "extern" declaration for "MPIU_trspace" in order to call it from your test program.
You can use the attached (untested against HEAD) patch to give you a hook for #2. You'll need to throw down an "extern" declaration for the routine in your test program as well.
-Dave
On Feb 22, 2013, at 9:28 AM CST, Jim Dinan wrote:
Jeff,
Which memory are you interested in -- memory the application allocates, or memory that MPICH allocates? If the former, you can use the PMPI interface to hook MPI_Alloc/Free_mem. If you are interested in MPICH internal memory use, Dave can probably give the most informed answer. I would probably just modify MPIU_Malloc.
~Jim.
On 2/22/13 8:44 AM, Jeff Hammond wrote:
Is there any documentation of how to properly instrument MPICH with memory profiling? I understand MPIU_MALLOC allows me to do this.
My goal is to merely print out every time memory is allocated larger than a particular size. I'd also like the rank to show up in the output.
Please RTM me if the M exists. Source comments are also a good M if they are correct and sufficiently complete.
Thanks!
Jeff
-- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute [email protected] / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond