Thanks for running the tests. I guess the next step, as we discussed at

UIUC, would be to use a lock per vc instead of lock per process and see if

that improves the message rate. (This simple change may not even work

because some other data structures may need to be protected, but it is worth

a try.)

Rajeev  

-----Original Message-----

From: owner-mpich2-core@mcs.anl.gov 

[mailto:owner-mpich2-core@mcs.anl.gov] On Behalf Of Pavan Balaji

Sent: Tuesday, January 29, 2008 3:53 PM

To: mpich2-core@mcs.anl.gov

Subject: [mpich2-core] MPICH2 Threading Overhead

Hi all,

I've run some experiments for the MPICH2 threading overhead 

on BG/P and Breadboard (test program is attached). I don't 

remember if IBM reported that the performance stays constant 

with increasing threads or if it degrades, but I'm noticing a 

drop in performance (results below).

On BG/P, this is slight yet noticeable, but on breadboard 

it's drastic. 

Note that 1 thread refers to "no extra threads" -- the main 

process does a self send/recv.

This is not the fairest comparison since MPICH2-BG/P is based 

on MPICH2-1.0.4p1, while the breadboard runs are on 

MPICH2-trunk. I'll try out MPICH2-1.0.4p1 on breadboard as 

well. But as a longer term solution I'm trying to get 

MPICH2-BG/P ported to MPICH2-trunk so that I can easily try 

out any threading enhancements that might go in to trunk from here on.

Thanks.

  -- Pavan

--------------------------------------------------------------

-----------

BG/P:

----

# MPI Message Rate Test with 1 Threads

# Size          Latency (us)    Message Rate (MMPS)

0               1.38            0.725

# MPI Message Rate Test with 2 Threads

# Size          Latency (us)    Message Rate (MMPS)

0               3.59            0.557

# MPI Message Rate Test with 3 Threads

# Size          Latency (us)    Message Rate (MMPS)

0               5.75            0.522

# MPI Message Rate Test with 4 Threads

# Size          Latency (us)    Message Rate (MMPS)

0               7.93            0.504

Breadboard:

----------

# MPI Message Rate Test with 1 Threads

# Size          Latency (us)    Message Rate (MMPS)

0               0.17            5.774

# MPI Message Rate Test with 2 Threads

# Size          Latency (us)    Message Rate (MMPS)

0               1.69            1.180

# MPI Message Rate Test with 3 Threads

# Size          Latency (us)    Message Rate (MMPS)

0               7.54            0.398

# MPI Message Rate Test with 4 Threads

# Size          Latency (us)    Message Rate (MMPS)

0               2.04            1.963

--

Pavan Balaji

http://www.mcs.anl.gov/~balaji