It would be great to run these tests. As we discussed, we should ignore for now that fact that a simple lock on each of the VCs is probably not entirely correct - however, it will give us some information on how much this approach might improve performance, at least on Breadboard. Bill On Jan 29, 2008, at 11:34 PM, Rajeev Thakur wrote:

Thanks for running the tests. I guess the next step, as we discussed at UIUC, would be to use a lock per vc instead of lock per process and see if that improves the message rate. (This simple change may not even work because some other data structures may need to be protected, but it is worth a try.)

Rajeev

...

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Tuesday, January 29, 2008 3:53 PM To: [email protected] Subject: [mpich2-core] MPICH2 Threading Overhead

Hi all,

I've run some experiments for the MPICH2 threading overhead on BG/P and Breadboard (test program is attached). I don't remember if IBM reported that the performance stays constant with increasing threads or if it degrades, but I'm noticing a drop in performance (results below).

On BG/P, this is slight yet noticeable, but on breadboard it's drastic. Note that 1 thread refers to "no extra threads" -- the main process does a self send/recv.

This is not the fairest comparison since MPICH2-BG/P is based on MPICH2-1.0.4p1, while the breadboard runs are on MPICH2-trunk. I'll try out MPICH2-1.0.4p1 on breadboard as well. But as a longer term solution I'm trying to get MPICH2-BG/P ported to MPICH2-trunk so that I can easily try out any threading enhancements that might go in to trunk from here on.

Thanks.

-- Pavan

-------------------------------------------------------------- -----------

BG/P: ---- # MPI Message Rate Test with 1 Threads # Size Latency (us) Message Rate (MMPS) 0 1.38 0.725 # MPI Message Rate Test with 2 Threads # Size Latency (us) Message Rate (MMPS) 0 3.59 0.557 # MPI Message Rate Test with 3 Threads # Size Latency (us) Message Rate (MMPS) 0 5.75 0.522 # MPI Message Rate Test with 4 Threads # Size Latency (us) Message Rate (MMPS) 0 7.93 0.504

Breadboard: ---------- # MPI Message Rate Test with 1 Threads # Size Latency (us) Message Rate (MMPS) 0 0.17 5.774 # MPI Message Rate Test with 2 Threads # Size Latency (us) Message Rate (MMPS) 0 1.69 1.180 # MPI Message Rate Test with 3 Threads # Size Latency (us) Message Rate (MMPS) 0 7.54 0.398 # MPI Message Rate Test with 4 Threads # Size Latency (us) Message Rate (MMPS) 0 2.04 1.963

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

What is the switch to turn on/off this optimization? Rajeev _____ From: [email protected] [mailto:[email protected]] On Behalf Of William Gropp Sent: Wednesday, January 30, 2008 8:30 AM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead Cool, this is great! This means that there's lots of room for improvements :) Is the one-thread case exploiting the "skip locks/nesting count through thread-specific storage" optimization? It would be good to run the 1-thread case with this optimization both on and off in any case, just to get a measure of the overhead. Bill On Jan 29, 2008, at 3:53 PM, Pavan Balaji wrote: Hi all, I've run some experiments for the MPICH2 threading overhead on BG/P and Breadboard (test program is attached). I don't remember if IBM reported that the performance stays constant with increasing threads or if it degrades, but I'm noticing a drop in performance (results below). On BG/P, this is slight yet noticeable, but on breadboard it's drastic. Note that 1 thread refers to "no extra threads" -- the main process does a self send/recv. This is not the fairest comparison since MPICH2-BG/P is based on MPICH2-1.0.4p1, while the breadboard runs are on MPICH2-trunk. I'll try out MPICH2-1.0.4p1 on breadboard as well. But as a longer term solution I'm trying to get MPICH2-BG/P ported to MPICH2-trunk so that I can easily try out any threading enhancements that might go in to trunk from here on. Thanks. -- Pavan ------------------------------------------------------------------------- BG/P: ---- # MPI Message Rate Test with 1 Threads # Size Latency (us) Message Rate (MMPS) 0 1.38 0.725 # MPI Message Rate Test with 2 Threads # Size Latency (us) Message Rate (MMPS) 0 3.59 0.557 # MPI Message Rate Test with 3 Threads # Size Latency (us) Message Rate (MMPS) 0 5.75 0.522 # MPI Message Rate Test with 4 Threads # Size Latency (us) Message Rate (MMPS) 0 7.93 0.504 Breadboard: ---------- # MPI Message Rate Test with 1 Threads # Size Latency (us) Message Rate (MMPS) 0 0.17 5.774 # MPI Message Rate Test with 2 Threads # Size Latency (us) Message Rate (MMPS) 0 1.69 1.180 # MPI Message Rate Test with 3 Threads # Size Latency (us) Message Rate (MMPS) 0 7.54 0.398 # MPI Message Rate Test with 4 Threads # Size Latency (us) Message Rate (MMPS) 0 2.04 1.963 -- Pavan Balaji http://www.mcs.anl.gov/~balaji #include "mpi.h" #include <stdio.h> #include <stdlib.h> #include <math.h> #include <unistd.h> #include <pthread.h> #include <assert.h> #include <string.h> #define REQUESTS 16 #define SKIP 100 #define LOOPS 1000000 #define MAX_THREADS 100 typedef struct { int nthreads; int size; } params_t; char * s_buf, * r_buf; int skip = 10; int loop = 10000; double t_start = 0.0, t_end = 0.0; pthread_barrier_t pbarrier; params_t params; void usage(char * progname) { fprintf(stderr, "Usage: %s -t [num_threads] -s [msg_size]\n", progname); MPI_Abort(MPI_COMM_WORLD, -1); } void get_parameters(int orig_argc, char ** orig_argv) { int argc = orig_argc; char ** argv = orig_argv; params.nthreads = -1; params.size = -1; while (--argc && ++argv) { if (!strcmp(*argv, "-t")) { --argc; ++argv; params.nthreads = atoi(*argv); continue; } if (!strcmp(*argv, "-s")) { --argc; ++argv; params.size = atoi(*argv); continue; } usage(orig_argv[0]); } if ((params.nthreads <= 0) || (params.size < 0)) usage(orig_argv[0]); } void * pingping_timed (void * arg) { int i, j; MPI_Request request[REQUESTS*2]; MPI_Status reqstat[REQUESTS*2]; pthread_barrier_wait(&pbarrier); for (i = 0; i < loop + skip; i++) { if (i == skip) t_start = MPI_Wtime(); for (j = 0; j < REQUESTS; j++) MPI_Irecv(r_buf, params.size, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &request[j+REQUESTS]); for (j = 0; j < REQUESTS; j++) MPI_Isend(s_buf, params.size, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &request[j]); MPI_Waitall(REQUESTS*2, request, reqstat); } pthread_barrier_wait(&pbarrier); t_end = MPI_Wtime(); return NULL; } int main(int argc, char ** argv) { int procs, i, pmode; pthread_t thread[MAX_THREADS]; void * retval[MAX_THREADS]; double latency, mrate; MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &pmode); if (pmode != MPI_THREAD_MULTIPLE) { fprintf(stderr, "Thread Multiple not supported by the MPI implementation\n"); MPI_Abort(MPI_COMM_WORLD, -1); } get_parameters(argc, argv); pthread_barrier_init(&pbarrier, NULL, params.nthreads); if (params.size) { s_buf = (void *) malloc(params.size); r_buf = (void *) malloc(params.size); for (i = 0; i < params.size; i++) { s_buf[i] = '0'; r_buf[i] = '0'; } } MPI_Comm_size(MPI_COMM_WORLD, &procs); if (procs != 1) { fprintf(stderr, "This is a 1 process test\n"); MPI_Abort(MPI_COMM_WORLD, -1); } MPI_Barrier(MPI_COMM_WORLD); for (i = 1; i < params.nthreads; i++) pthread_create(&thread[i], NULL, (void*) pingping_timed, (void *) &i); pingping_timed(NULL); for (i = 1; i < params.nthreads; i++) pthread_join(thread[i], &retval[i]); printf("# MPI Message Rate Test with %d Threads\n", params.nthreads); printf("# Size\t\tLatency (us)\tMessage Rate (MMPS) \n"); /* Latency is the time to send and recieve a message divided by two. */ latency = (t_end - t_start) * 1.0e6 / (2*loop*REQUESTS); mrate = params.nthreads / latency; printf("%d\t\t%0.2f\t\t%0.3f\n", params.size, latency, mrate); MPI_Finalize(); return 0; } William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

That should be enough - if the single thread case is started with MPI_THREAD_FUNNELED instead of MPI_THREAD_MULTIPLE, the code *should* use the runtime test to see if it needs to use thread locks and thread-private storage or not. Though it isn't pure MPI, moving the get_parameters call ahead of MPI_Init_thread and using MPI_THREAD_FUNNELED if the number of threads is 1 would work; it would be even better to have an option to use thread_multiple in that case, just to make it easier to run all versions of the test. Bill On Jan 30, 2008, at 10:44 AM, Darius Buntinas wrote:

Don't you just NOT call MPI_thread_init?

-d

On 01/30/2008 10:43 AM, Rajeev Thakur wrote:

...

What is the switch to turn on/off this optimization? Rajeev

--------------------------------------------------------------------- --- *From:* [email protected] [mailto:[email protected]] *On Behalf Of *William Gropp *Sent:* Wednesday, January 30, 2008 8:30 AM *To:* [email protected] *Subject:* Re: [mpich2-core] MPICH2 Threading Overhead Cool, this is great! This means that there's lots of room for improvements :) Is the one-thread case exploiting the "skip locks/nesting count through thread-specific storage" optimization? It would be good to run the 1-thread case with this optimization both on and off in any case, just to get a measure of the overhead. Bill On Jan 29, 2008, at 3:53 PM, Pavan Balaji wrote:

...

Hi all,

I've run some experiments for the MPICH2 threading overhead on BG/P and Breadboard (test program is attached). I don't remember if IBM reported that the performance stays constant with increasing threads or if it degrades, but I'm noticing a drop in performance (results below).

On BG/P, this is slight yet noticeable, but on breadboard it's drastic. Note that 1 thread refers to "no extra threads" -- the main process does a self send/recv.

This is not the fairest comparison since MPICH2-BG/P is based on MPICH2-1.0.4p1, while the breadboard runs are on MPICH2-trunk. I'll try out MPICH2-1.0.4p1 on breadboard as well. But as a longer term solution I'm trying to get MPICH2-BG/P ported to MPICH2- trunk so that I can easily try out any threading enhancements that might go in to trunk from here on.

Thanks.

-- Pavan

-------------------------------------------------------------------- -----

BG/P: ---- # MPI Message Rate Test with 1 Threads # Size Latency (us) Message Rate (MMPS) 0 1.38 0.725 # MPI Message Rate Test with 2 Threads # Size Latency (us) Message Rate (MMPS) 0 3.59 0.557 # MPI Message Rate Test with 3 Threads # Size Latency (us) Message Rate (MMPS) 0 5.75 0.522 # MPI Message Rate Test with 4 Threads # Size Latency (us) Message Rate (MMPS) 0 7.93 0.504

Breadboard: ---------- # MPI Message Rate Test with 1 Threads # Size Latency (us) Message Rate (MMPS) 0 0.17 5.774 # MPI Message Rate Test with 2 Threads # Size Latency (us) Message Rate (MMPS) 0 1.69 1.180 # MPI Message Rate Test with 3 Threads # Size Latency (us) Message Rate (MMPS) 0 7.54 0.398 # MPI Message Rate Test with 4 Threads # Size Latency (us) Message Rate (MMPS) 0 2.04 1.963

-- Pavan Balaji http://www.mcs.anl.gov/~balaji #include "mpi.h" #include <stdio.h> #include <stdlib.h> #include <math.h> #include <unistd.h> #include <pthread.h> #include <assert.h> #include <string.h>

#define REQUESTS 16 #define SKIP 100 #define LOOPS 1000000 #define MAX_THREADS 100

typedef struct { int nthreads; int size; } params_t;

char * s_buf, * r_buf; int skip = 10; int loop = 10000; double t_start = 0.0, t_end = 0.0; pthread_barrier_t pbarrier; params_t params;

void usage(char * progname) { fprintf(stderr, "Usage: %s -t [num_threads] -s [msg_size]\n", progname); MPI_Abort(MPI_COMM_WORLD, -1); }

void get_parameters(int orig_argc, char ** orig_argv) { int argc = orig_argc; char ** argv = orig_argv;

params.nthreads = -1; params.size = -1;

while (--argc && ++argv) { if (!strcmp(*argv, "-t")) { --argc; ++argv; params.nthreads = atoi(*argv); continue; } if (!strcmp(*argv, "-s")) { --argc; ++argv; params.size = atoi(*argv); continue; }

usage(orig_argv[0]); }

if ((params.nthreads <= 0) || (params.size < 0)) usage(orig_argv[0]); }

void * pingping_timed (void * arg) { int i, j; MPI_Request request[REQUESTS*2]; MPI_Status reqstat[REQUESTS*2];

pthread_barrier_wait(&pbarrier); for (i = 0; i < loop + skip; i++) { if (i == skip) t_start = MPI_Wtime();

for (j = 0; j < REQUESTS; j++) MPI_Irecv(r_buf, params.size, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &request[j+REQUESTS]);

for (j = 0; j < REQUESTS; j++) MPI_Isend(s_buf, params.size, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &request[j]);

MPI_Waitall(REQUESTS*2, request, reqstat); } pthread_barrier_wait(&pbarrier);

t_end = MPI_Wtime();

return NULL; }

int main(int argc, char ** argv) { int procs, i, pmode; pthread_t thread[MAX_THREADS]; void * retval[MAX_THREADS]; double latency, mrate;

MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &pmode); if (pmode != MPI_THREAD_MULTIPLE) { fprintf(stderr, "Thread Multiple not supported by the MPI implementation\n"); MPI_Abort(MPI_COMM_WORLD, -1); }

get_parameters(argc, argv); pthread_barrier_init(&pbarrier, NULL, params.nthreads);

if (params.size) { s_buf = (void *) malloc(params.size); r_buf = (void *) malloc(params.size); for (i = 0; i < params.size; i++) { s_buf[i] = '0'; r_buf[i] = '0'; } }

MPI_Comm_size(MPI_COMM_WORLD, &procs); if (procs != 1) { fprintf(stderr, "This is a 1 process test\n"); MPI_Abort(MPI_COMM_WORLD, -1); } MPI_Barrier(MPI_COMM_WORLD);

for (i = 1; i < params.nthreads; i++) pthread_create(&thread[i], NULL, (void*) pingping_timed, (void *) &i);

pingping_timed(NULL);

for (i = 1; i < params.nthreads; i++) pthread_join(thread[i], &retval[i]);

printf("# MPI Message Rate Test with %d Threads\n", params.nthreads); printf("# Size\t\tLatency (us)\tMessage Rate (MMPS) \n");

/* Latency is the time to send and recieve a message divided by two. */ latency = (t_end - t_start) * 1.0e6 / (2*loop*REQUESTS); mrate = params.nthreads / latency; printf("%d\t\t%0.2f\t\t%0.3f\n", params.size, latency, mrate);

MPI_Finalize();

return 0; }

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

It should be more visible with MPICH2 on breadboard than IBM's MPI. Don't know what they do. Rajeev

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Wednesday, January 30, 2008 11:09 AM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

I tried it on BG/P, and the performance does seem to improve a little bit -- from 0.725 MMPS to 0.795 MMPS.

-- Pavan

On 01/30/2008 10:52 AM, William Gropp wrote:

...

That should be enough - if the single thread case is started with MPI_THREAD_FUNNELED instead of MPI_THREAD_MULTIPLE, the code *should* use the runtime test to see if it needs to use thread locks and thread-private storage or not.

Though it isn't pure MPI, moving the get_parameters call ahead of MPI_Init_thread and using MPI_THREAD_FUNNELED if the number of threads is 1 would work; it would be even better to have an option to use thread_multiple in that case, just to make it easier to run all versions of the test.

Bill

On Jan 30, 2008, at 10:44 AM, Darius Buntinas wrote:

...

Don't you just NOT call MPI_thread_init?

-d

On 01/30/2008 10:43 AM, Rajeev Thakur wrote:

...

What is the switch to turn on/off this optimization? Rajeev

---

----------

...

...

...

*From:* [email protected] <mailto:[email protected]> [mailto:[email protected]] *On Behalf Of

*William Gropp

...

*Sent:* Wednesday, January 30, 2008 8:30 AM *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [mpich2-core] MPICH2 Threading Overhead Cool, this is great! This means that there's lots of room for improvements :) Is the one-thread case exploiting the "skip

locks/nesting count

...

through thread-specific storage" optimization? It

would be good to

...

run the 1-thread case with this optimization both on

and off in any

...

case, just to get a measure of the overhead. Bill On Jan 29, 2008, at 3:53 PM, Pavan Balaji wrote:

...

Hi all,

I've run some experiments for the MPICH2 threading

overhead on

...

...

BG/P and Breadboard (test program is attached). I

don't remember

...

if IBM reported that the performance stays constant with increasing threads or if it degrades, but I'm

noticing a drop in

...

performance (results below).

On BG/P, this is slight yet noticeable, but on

breadboard it's

...

drastic. Note that 1 thread refers to "no extra

threads" -- the

...

main process does a self send/recv.

This is not the fairest comparison since MPICH2-BG/P

is based on

...

MPICH2-1.0.4p1, while the breadboard runs are on

MPICH2-trunk.

...

I'll try out MPICH2-1.0.4p1 on breadboard as well.

But as a longer

...

term solution I'm trying to get MPICH2-BG/P ported

to MPICH2-trunk

...

so that I can easily try out any threading

enhancements that might

...

go in to trunk from here on.

Thanks.

-- Pavan

---

-----------

...

...

...

...

BG/P: ---- # MPI Message Rate Test with 1 Threads # Size Latency (us) Message Rate (MMPS) 0 1.38 0.725 # MPI Message Rate Test with 2 Threads # Size Latency (us) Message Rate (MMPS) 0 3.59 0.557 # MPI Message Rate Test with 3 Threads # Size Latency (us) Message Rate (MMPS) 0 5.75 0.522 # MPI Message Rate Test with 4 Threads # Size Latency (us) Message Rate (MMPS) 0 7.93 0.504

Breadboard: ---------- # MPI Message Rate Test with 1 Threads # Size Latency (us) Message Rate (MMPS) 0 0.17 5.774 # MPI Message Rate Test with 2 Threads # Size Latency (us) Message Rate (MMPS) 0 1.69 1.180 # MPI Message Rate Test with 3 Threads # Size Latency (us) Message Rate (MMPS) 0 7.54 0.398 # MPI Message Rate Test with 4 Threads # Size Latency (us) Message Rate (MMPS) 0 2.04 1.963

-- Pavan Balaji http://www.mcs.anl.gov/~balaji #include "mpi.h" #include <stdio.h> #include <stdlib.h> #include <math.h> #include <unistd.h> #include <pthread.h> #include <assert.h> #include <string.h>

#define REQUESTS 16 #define SKIP 100 #define LOOPS 1000000 #define MAX_THREADS 100

typedef struct { int nthreads; int size; } params_t;

char * s_buf, * r_buf; int skip = 10; int loop = 10000; double t_start = 0.0, t_end = 0.0; pthread_barrier_t pbarrier; params_t params;

void usage(char * progname) { fprintf(stderr, "Usage: %s -t [num_threads] -s [msg_size]\n", progname); MPI_Abort(MPI_COMM_WORLD, -1); }

void get_parameters(int orig_argc, char ** orig_argv) { int argc = orig_argc; char ** argv = orig_argv;

params.nthreads = -1; params.size = -1;

while (--argc && ++argv) { if (!strcmp(*argv, "-t")) { --argc; ++argv; params.nthreads = atoi(*argv); continue; } if (!strcmp(*argv, "-s")) { --argc; ++argv; params.size = atoi(*argv); continue; }

usage(orig_argv[0]); }

if ((params.nthreads <= 0) || (params.size < 0)) usage(orig_argv[0]); }

void * pingping_timed (void * arg) { int i, j; MPI_Request request[REQUESTS*2]; MPI_Status reqstat[REQUESTS*2];

pthread_barrier_wait(&pbarrier); for (i = 0; i < loop + skip; i++) { if (i == skip) t_start = MPI_Wtime();

for (j = 0; j < REQUESTS; j++) MPI_Irecv(r_buf, params.size, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &request[j+REQUESTS]);

for (j = 0; j < REQUESTS; j++) MPI_Isend(s_buf, params.size, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &request[j]);

MPI_Waitall(REQUESTS*2, request, reqstat); } pthread_barrier_wait(&pbarrier);

t_end = MPI_Wtime();

return NULL; }

int main(int argc, char ** argv) { int procs, i, pmode; pthread_t thread[MAX_THREADS]; void * retval[MAX_THREADS]; double latency, mrate;

MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &pmode); if (pmode != MPI_THREAD_MULTIPLE) { fprintf(stderr, "Thread Multiple not supported by the MPI implementation\n"); MPI_Abort(MPI_COMM_WORLD, -1); }

get_parameters(argc, argv); pthread_barrier_init(&pbarrier, NULL, params.nthreads);

if (params.size) { s_buf = (void *) malloc(params.size); r_buf = (void *) malloc(params.size); for (i = 0; i < params.size; i++) { s_buf[i] = '0'; r_buf[i] = '0'; } }

MPI_Comm_size(MPI_COMM_WORLD, &procs); if (procs != 1) { fprintf(stderr, "This is a 1 process test\n"); MPI_Abort(MPI_COMM_WORLD, -1); } MPI_Barrier(MPI_COMM_WORLD);

for (i = 1; i < params.nthreads; i++) pthread_create(&thread[i], NULL, (void*)

pingping_timed, (void

...

*) &i);

pingping_timed(NULL);

for (i = 1; i < params.nthreads; i++) pthread_join(thread[i], &retval[i]);

printf("# MPI Message Rate Test with %d Threads\n", params.nthreads); printf("# Size\t\tLatency (us)\tMessage Rate (MMPS) \n");

/* Latency is the time to send and recieve a message divided by two. */ latency = (t_end - t_start) * 1.0e6 / (2*loop*REQUESTS); mrate = params.nthreads / latency; printf("%d\t\t%0.2f\t\t%0.3f\n", params.size, latency, mrate);

MPI_Finalize();

return 0; } William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

Rajeev, It's the same test. I just cleaned out some stuff and made it easier to pass command line options. Do you see any major difference? -- Pavan On 01/30/2008 11:14 AM, Rajeev Thakur wrote:

Pavan, Where did you get this threads.c test? The messaging rate benchmark that Sameer had sent is different. See attached mail.

Rajeev

...

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Tuesday, January 29, 2008 3:53 PM To: [email protected] Subject: [mpich2-core] MPICH2 Threading Overhead

Hi all,

I've run some experiments for the MPICH2 threading overhead on BG/P and Breadboard (test program is attached). I don't remember if IBM reported that the performance stays constant with increasing threads or if it degrades, but I'm noticing a drop in performance (results below).

On BG/P, this is slight yet noticeable, but on breadboard it's drastic. Note that 1 thread refers to "no extra threads" -- the main process does a self send/recv.

This is not the fairest comparison since MPICH2-BG/P is based on MPICH2-1.0.4p1, while the breadboard runs are on MPICH2-trunk. I'll try out MPICH2-1.0.4p1 on breadboard as well. But as a longer term solution I'm trying to get MPICH2-BG/P ported to MPICH2-trunk so that I can easily try out any threading enhancements that might go in to trunk from here on.

Thanks.

-- Pavan

-------------------------------------------------------------- -----------

BG/P: ---- # MPI Message Rate Test with 1 Threads # Size Latency (us) Message Rate (MMPS) 0 1.38 0.725 # MPI Message Rate Test with 2 Threads # Size Latency (us) Message Rate (MMPS) 0 3.59 0.557 # MPI Message Rate Test with 3 Threads # Size Latency (us) Message Rate (MMPS) 0 5.75 0.522 # MPI Message Rate Test with 4 Threads # Size Latency (us) Message Rate (MMPS) 0 7.93 0.504

Breadboard: ---------- # MPI Message Rate Test with 1 Threads # Size Latency (us) Message Rate (MMPS) 0 0.17 5.774 # MPI Message Rate Test with 2 Threads # Size Latency (us) Message Rate (MMPS) 0 1.69 1.180 # MPI Message Rate Test with 3 Threads # Size Latency (us) Message Rate (MMPS) 0 7.54 0.398 # MPI Message Rate Test with 4 Threads # Size Latency (us) Message Rate (MMPS) 0 2.04 1.963

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

------------------------------------------------------------------------

Subject: Messaging rate benchmark From: "Sameer Kumar" <[email protected]> Date: Tue, 25 Sep 2007 12:46:39 -0600 To: "Rajeev Thakur" <[email protected]>

To: "Rajeev Thakur" <[email protected]> CC: "'Pete Beckman'" <[email protected]>, "Craig Stunkel" <[email protected]>, "'William Gropp'" <[email protected]>, "Robert Wisniewski" <[email protected]>, "Gabor Dozsa" <[email protected]>

Hi, The message rate benchmark is attached. To compile

mpicc -o mrate-smp mrate-smp.c -lpthread

Please let me know if you have any problems running it. The number of threads to simulteneous inject messages is a runtime commandline parameter to the program. It seems to work fine on BG/P.

(See attached file: mrate-smp.c)

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

Ah! Of course. I hate thread programming :) All variables should be thread private unless otherwise annotated. Bill PS - It would still be good to run the test a few times - if the results are consistent, then all is well, if not, further upgrades are possible. On Jan 30, 2008, at 12:31 PM, Darius Buntinas wrote:

The 170ns number is probably due to a race we found. Each thread writes to start_time and end_time. We'll fix the program and try again.

-d

On 01/30/2008 11:50 AM, William Gropp wrote:

...

BTW, the Breadboard results suggest that the timing program is too simple in this environment. It would be good for the timing program to run the same test multiple times and at least report the spread in the results. Do we even believe the 170ns latency time for the 1-thread case? If so, is that the new record? (Note also that the test violates the same-send/recv-buffer rule :) ) Bill On Jan 29, 2008, at 3:53 PM, Pavan Balaji wrote:

...

Hi all,

I've run some experiments for the MPICH2 threading overhead on BG/ P and Breadboard (test program is attached). I don't remember if IBM reported that the performance stays constant with increasing threads or if it degrades, but I'm noticing a drop in performance (results below).

On BG/P, this is slight yet noticeable, but on breadboard it's drastic. Note that 1 thread refers to "no extra threads" -- the main process does a self send/recv.

This is not the fairest comparison since MPICH2-BG/P is based on MPICH2-1.0.4p1, while the breadboard runs are on MPICH2-trunk. I'll try out MPICH2-1.0.4p1 on breadboard as well. But as a longer term solution I'm trying to get MPICH2-BG/P ported to MPICH2-trunk so that I can easily try out any threading enhancements that might go in to trunk from here on.

Thanks.

-- Pavan

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

I think we should worry about the locking overhead alone and not dilute the benchmark by adding network communication in it. So, each process sending to itself would be better, IMO. Btw, there was a suggestion earlier that we should try to place one lock per VC to see if the performance improves. However, I think we are jumping too far ahead with that. Even with a global lock, I would expect that the performance would stay constant (note that ideal behavior is linear increase in messaging rate with number of threads). However, it's dropping currently. I feel that the priority order should be: 1. Instrument the code for overheads with the global lock that is causing the performance to drop. 2. Instrument the hardware -- we never ran SMP mode BG/P with 4 processes (not threads) doing the messaging rate test separately. 3. Port MPICH2/BG to trunk so that we can try out new enhancements on both breadboard and BG/P. Thoughts? -- Pavan On 01/30/2008 01:07 PM, Darius Buntinas wrote:

I just want to check what the benchmark should do.

My understanding from the conference calls and Sameer's slides is that they're interested in the case where multiple threads of a process are sending/receiving to/from different processes. This allows the process to use multiple DMAs concurrently.

Right now, we've been testing the case where one process is sending/receiving with itself. So moving the locks around VCs may decrease the size of the CS, but will still serialize.

So what's the test we want to run?

I imagine something like P processes each with P-1 threads where each thread is communicating with a different process. But then we need a machine with P(P-1) cores.

On the 8-core nodes we have, we can run 3 processes with 2 threads each. But I think we need more threads to really stress things.

It seems that what we can see from the one or two processes is the just the effect of the size of the CS.

Comments?

-d

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

I think we should worry about the locking overhead alone and not dilute the benchmark by adding network communication in it. So, each process sending to itself would be better, IMO.

I think we want to know how much is the locking overhead in the presence of network communication because that is what happens in reality.

Btw, there was a suggestion earlier that we should try to place one lock per VC to see if the performance improves. However, I think we are jumping too far ahead with that. Even with a global lock, I would expect that the performance would stay constant (note that ideal behavior is linear increase in messaging rate with number of threads). However, it's dropping currently.

Why would the performance stay constant with the global lock?

1. Instrument the code for overheads with the global lock that is causing the performance to drop.

2. Instrument the hardware -- we never ran SMP mode BG/P with 4 processes (not threads) doing the messaging rate test separately.

3. Port MPICH2/BG to trunk so that we can try out new enhancements on both breadboard and BG/P.

I think Sameer would like us to hack IBM's MPI implementation directly. That would save us the porting time as well (for now). Rajeev

...

I think we want to know how much is the locking overhead in the presence of network communication because that is what happens in reality.

Yeah, but if DCMF or the DMA engines have some overhead in threading, shouldn't we try to avoid it? I was trying to avoid the amount of code in non-MPICH2 part of the stack that is utilized.

Once we fix the MPICH2 part, it of course makes sense to measure the overall stack. But for the time being, I think we should not dilute the test too much.

May be we should run both versions of the test then.

...

Why would the performance stay constant with the global lock?

If only one thread is allowed to compute at any point of time, the performance should be the same as just having one thread, isn't it? This is the ideal case -- we might not achieve it if the locks are too expensive, for example. But we should know what is causing the overhead and see if we can avoid it.

Well, there is a lot more communication going on in the multithreaded case, and that is not being accounted for in the average time calculation.

...

I think Sameer would like us to hack IBM's MPI implementation directly. That would save us the porting time as well (for now).

Maintaining two stacks is a pain. Also, IBM's MPI implementation is MPICH2.

In the long run yes, but in the very short run we want to show that we are touching their code, and that their effort in opening it was worthwhile. Their purpose in opening it is that they will get contributions back from the community. (There has been some separate discussion about them wanting the community to tune the collective communication routines that they haven't, e.g. the scatter/bcast performance problem I told them about.) Rajeev

Hmm.. You lost me. Are you saying that the multi-threaded case with 4 threads has more than 4X communication as compared to the single threaded case?

Each thread calls Isend-Irecv 160,000 times. With 4 threads, there will be 640,000 calls. There is a barrier across the entire set. But there is no division by 4 in the time calculation. Rajeev

Ah, it's taken care of in the message rate calculation. I missed that. Rajeev

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Wednesday, January 30, 2008 3:47 PM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

So the latency will increase by 4 times, but why will the messaging rate decrease?

-- Pavan

On 01/30/2008 03:45 PM, Rajeev Thakur wrote:

...

...

Hmm.. You lost me. Are you saying that the multi-threaded case with 4 threads has more than 4X communication as compared to the single threaded case?

Each thread calls Isend-Irecv 160,000 times. With 4 threads, there will be 640,000 calls. There is a barrier across the entire set. But there is no division by 4 in the time calculation.

Rajeev

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

Much as I'd love the instrumentation, at least some of the tests can be done with just a few changes. Currently, most of the code looks like this (in the proc-null mode): get global lock get thread-private storage, increment nest count ... do stuff (which assumes that the global lock is held) decrement nest count from thread-private storage release global lock We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests. We can answer a version of (2) by changing the increment/decrement macros to acquire/release the global lock. (3) will require inspection (and reorganization) of the code, following the work that Rajeev and I did. Bill On Jan 31, 2008, at 8:00 AM, Pavan Balaji wrote:

Below are the numbers while using MPI_PROC_NULL. The trend is the same. I guess the next step is to start instrumenting the code.

BG/P: ---- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 1.02 0.981 2 0 1.21 0.825 3 0 2.48 0.403 4 0 3.78 0.265

Breadboard: ---------- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.39 2.589 2 0 0.85 1.183 3 0 4.68 0.213 4 0 4.36 0.229

Thanks.

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

Thanks. Note that in (2), the idea is to not acquire the global lock in the func enter/exit; just as needed for the atomic updates (these aren't the only places that a local atomic op would be needed; the object allocator is another one and that isn't fine-grain ready). But for the proc-null test, it may be ok. My hypothesis is that (1) is a relatively small cost compared to the big lock and that (2) will give some improvement but at the cost of increased latency. As another goal (unrelated to threads), we should try to get the proc- null latency well under 1us on BG/P. Is the 1 thread case with THREAD_MULTIPLE or THREAD_FUNNELED? Bill On Jan 31, 2008, at 9:00 AM, Pavan Balaji wrote:

I'll try out (1) and (2) and see if it works. Given that the lock contention in our test program is so high, there is a high likelihood that it might not work at all.

-- Pavan

On 01/31/2008 08:38 AM, William Gropp wrote:

...

Much as I'd love the instrumentation, at least some of the tests can be done with just a few changes. Currently, most of the code looks like this (in the proc-null mode): get global lock get thread-private storage, increment nest count ... do stuff (which assumes that the global lock is held) decrement nest count from thread-private storage release global lock We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests. We can answer a version of (2) by changing the increment/decrement macros to acquire/release the global lock. (3) will require inspection (and reorganization) of the code, following the work that Rajeev and I did. Bill On Jan 31, 2008, at 8:00 AM, Pavan Balaji wrote:

...

Below are the numbers while using MPI_PROC_NULL. The trend is the same. I guess the next step is to start instrumenting the code.

BG/P: ---- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 1.02 0.981 2 0 1.21 0.825 3 0 2.48 0.403 4 0 3.78 0.265

Breadboard: ---------- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.39 2.589 2 0 0.85 1.183 3 0 4.68 0.213 4 0 4.36 0.229

Thanks.

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

Ah, I had missed the fact that it didn't scale the results. So is the cost of the MPI call stack to the top of the ch3 code (where the proc-null test is) about 25ns on Breadboard and 60ns on BG/P? Bill On Jan 31, 2008, at 9:25 AM, Pavan Balaji wrote:

Bill,

...

As another goal (unrelated to threads), we should try to get the proc-null latency well under 1us on BG/P.

Note that the latency I sent is not regular ping-pong latency. The test does 16 Isends and 16 Irecvs, and a waitall. This is the way Sameer's original test was written, so I didn't want to modify that.

...

Is the 1 thread case with THREAD_MULTIPLE or THREAD_FUNNELED?

The single thread case doesn't use MPI_Init_thread at all.

Thanks.

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

Ok, so the latencies for the MPI call stack, even without the thread locks, is greater than we'd like. To get the desired message rate, we'll also need to improve that. Do these tests turn off the error checking? Would it be possible to run the tests with a return instead of a function call at various levels? Since the operations are basically no-ops, we (ie, Pavan) could then easily measure the contribution of each layer, without worrying about the cost of instrumentation. (This is less important than getting some more thread results, but it will be helpful in guiding any more substantive code changes). Bill On Jan 31, 2008, at 9:36 AM, Pavan Balaji wrote:

...

Ah, I had missed the fact that it didn't scale the results. So is the cost of the MPI call stack to the top of the ch3 code (where the proc-null test is) about 25ns on Breadboard and 60ns on BG/P?

No. The latency number is already total_time / (LOOPS * 16), so you can't divide the latency number by 16 again.

Sorry for the confusion.

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

Bill,

Thanks. Note that in (2), the idea is to not acquire the global lock in the func enter/exit; just as needed for the atomic updates (these aren't the only places that a local atomic op would be needed; the object allocator is another one and that isn't fine-grain ready). But for the proc-null test, it may be ok.

How will acquiring a global lock in the increment and releasing it in the decrement be different from the current global lock? These are anyway done at the same time currently for embedded MPI calls. Or did I misunderstand what you meant? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

Bill,

We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain?

We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.

Do you mean acquire the global lock in the nest count increment? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

Bill, The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now. -- Pavan On 01/31/2008 01:17 PM, William Gropp wrote:

No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency.

Bill

On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:

...

Bill,

...

We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.

Do you mean acquire the global lock in the nest count increment?

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

For MPICH2 over BG/P, there is no way to control the nesting counters from within MPI. The DCMF layer is handling this. -- Pavan On 01/31/2008 02:55 PM, William Gropp wrote:

The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated.

Bill

On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:

...

Bill,

The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.

-- Pavan

On 01/31/2008 01:17 PM, William Gropp wrote:

...

No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency. Bill On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:

...

Bill,

...

We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.

Do you mean acquire the global lock in the nest count increment?

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

I removed the send and receive calls from the timing loop and replaced them with statements that increment and decrement a volatile variable. The volatile variable is in an array, so each thread accesses its own variable. The array elements are also padded so each is in its own cache line. So there should be no contention between threads on the variable. It looks like the test program is affecting the results. :-) # MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 306.547 2 0 0.00 483.213 3 0 0.01 184.908 4 0 0.01 114.825 5 0 0.01 78.911 6 0 0.02 59.759 7 0 0.02 47.333 8 0 0.03 38.067 -d On 01/31/2008 06:02 PM, Darius Buntinas wrote:

I removed the locks and nesting code (i.e., defined CS_ENTER/EXIT to nothing). I then changed the test to use MPI_Send and MPI_Recv (instead of isend and irecv), so I wouldn't have to use a lock when allocating a request. I then ran it on intel-loaner1 (dual quadcore) sending and receiving to MPI_PROC_NULL. Below are the results (first with locking intact, then with the locking removed).

There's an improvement in message rate, but we still see message rate decrease as we add additional threads. What's interesting, is that even the single threaded case (that doesn't call MPI_Init_thread) shows improvement with locks removed. I can't explain that, since it's not using locks (or at least shouldn't). I'll look into this further.

-d

CONTROL CASE WITH LOCKING INTACT: # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.15 6.774 2 0 0.57 1.755 3 0 1.29 0.773 4 0 2.07 0.483 5 0 2.73 0.366 6 0 3.12 0.321 7 0 3.45 0.290 8 0 3.69 0.271

LOCKING REMOVED: % ./threads-2 -t 8 -l 10000 -pn # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.09 11.709 2 0 0.11 9.128 3 0 0.20 5.032 4 0 0.26 3.779 5 0 0.32 3.158 6 0 0.38 2.635 7 0 0.45 2.243 8 0 0.50 1.989

On 01/31/2008 02:55 PM, William Gropp wrote:

...

The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated.

Bill

On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:

...

Bill,

The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.

-- Pavan

On 01/31/2008 01:17 PM, William Gropp wrote:

...

No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency. Bill On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:

...

Bill,

...

We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.

Do you mean acquire the global lock in the nest count increment?

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

Sure, it's on bblogin ~buntinas/ng-mpich/mrate/test/threads-2.c . It's based on your threads.c that I grabbed from your directory around 4:00. I moved the pthread_barriers outside the timing loop and ran it again. The "message rate" improved, but still decreased as the number of threads increased. -d On 01/31/2008 06:24 PM, Pavan Balaji wrote:

Very weird. Can you send me your test program? I'd like to poke into it a bit.

-- Pavan

On 01/31/2008 06:21 PM, Darius Buntinas wrote:

...

I removed the send and receive calls from the timing loop and replaced them with statements that increment and decrement a volatile variable. The volatile variable is in an array, so each thread accesses its own variable. The array elements are also padded so each is in its own cache line. So there should be no contention between threads on the variable.

It looks like the test program is affecting the results. :-)

# MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 306.547 2 0 0.00 483.213 3 0 0.01 184.908 4 0 0.01 114.825 5 0 0.01 78.911 6 0 0.02 59.759 7 0 0.02 47.333 8 0 0.03 38.067

-d

On 01/31/2008 06:02 PM, Darius Buntinas wrote:

...

I removed the locks and nesting code (i.e., defined CS_ENTER/EXIT to nothing). I then changed the test to use MPI_Send and MPI_Recv (instead of isend and irecv), so I wouldn't have to use a lock when allocating a request. I then ran it on intel-loaner1 (dual quadcore) sending and receiving to MPI_PROC_NULL. Below are the results (first with locking intact, then with the locking removed).

There's an improvement in message rate, but we still see message rate decrease as we add additional threads. What's interesting, is that even the single threaded case (that doesn't call MPI_Init_thread) shows improvement with locks removed. I can't explain that, since it's not using locks (or at least shouldn't). I'll look into this further.

-d

CONTROL CASE WITH LOCKING INTACT: # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.15 6.774 2 0 0.57 1.755 3 0 1.29 0.773 4 0 2.07 0.483 5 0 2.73 0.366 6 0 3.12 0.321 7 0 3.45 0.290 8 0 3.69 0.271

LOCKING REMOVED: % ./threads-2 -t 8 -l 10000 -pn # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.09 11.709 2 0 0.11 9.128 3 0 0.20 5.032 4 0 0.26 3.779 5 0 0.32 3.158 6 0 0.38 2.635 7 0 0.45 2.243 8 0 0.50 1.989

On 01/31/2008 02:55 PM, William Gropp wrote:

...

The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated.

Bill

On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:

...

Bill,

The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.

-- Pavan

On 01/31/2008 01:17 PM, William Gropp wrote:

...

No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency. Bill On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:

...

Bill,

...

We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.

Do you mean acquire the global lock in the nest count increment?

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

Holy googly moogly! The OS schedules each process on a different core. I forced it to schedule everything on one core and the performance suddenly shot up. Here are the numbers: /* Round robin scheduling of processes to cores */ #bb09# /home/balaji> mpiexec -n 1 ./threads -t 4 -l 100000 -r # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.13 7.899 2 0 1.12 1.780 3 0 3.14 0.955 4 0 4.59 0.871 /* Scheduling all processes on one core */ #bb09# /home/balaji> mpiexec -n 1 ./threads -t 4 -l 100000 # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.13 7.855 2 0 0.26 7.819 3 0 0.38 7.826 4 0 0.51 7.768 -- Pavan On 01/31/2008 07:17 PM, William Gropp wrote:

At what point does the OS decide to schedule these threads on different cores? In the classic multithreaded GUI application, the best strategy is to schedule all threads on the same core.

Bill

On Jan 31, 2008, at 6:58 PM, Darius Buntinas wrote:

...

Sure, it's on bblogin ~buntinas/ng-mpich/mrate/test/threads-2.c . It's based on your threads.c that I grabbed from your directory around 4:00.

I moved the pthread_barriers outside the timing loop and ran it again. The "message rate" improved, but still decreased as the number of threads increased.

-d

On 01/31/2008 06:24 PM, Pavan Balaji wrote:

...

Very weird. Can you send me your test program? I'd like to poke into it a bit. -- Pavan On 01/31/2008 06:21 PM, Darius Buntinas wrote:

...

I removed the send and receive calls from the timing loop and replaced them with statements that increment and decrement a volatile variable. The volatile variable is in an array, so each thread accesses its own variable. The array elements are also padded so each is in its own cache line. So there should be no contention between threads on the variable.

It looks like the test program is affecting the results. :-)

# MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 306.547 2 0 0.00 483.213 3 0 0.01 184.908 4 0 0.01 114.825 5 0 0.01 78.911 6 0 0.02 59.759 7 0 0.02 47.333 8 0 0.03 38.067

-d

On 01/31/2008 06:02 PM, Darius Buntinas wrote:

...

I removed the locks and nesting code (i.e., defined CS_ENTER/EXIT to nothing). I then changed the test to use MPI_Send and MPI_Recv (instead of isend and irecv), so I wouldn't have to use a lock when allocating a request. I then ran it on intel-loaner1 (dual quadcore) sending and receiving to MPI_PROC_NULL. Below are the results (first with locking intact, then with the locking removed).

There's an improvement in message rate, but we still see message rate decrease as we add additional threads. What's interesting, is that even the single threaded case (that doesn't call MPI_Init_thread) shows improvement with locks removed. I can't explain that, since it's not using locks (or at least shouldn't). I'll look into this further.

-d

CONTROL CASE WITH LOCKING INTACT: # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.15 6.774 2 0 0.57 1.755 3 0 1.29 0.773 4 0 2.07 0.483 5 0 2.73 0.366 6 0 3.12 0.321 7 0 3.45 0.290 8 0 3.69 0.271

LOCKING REMOVED: % ./threads-2 -t 8 -l 10000 -pn # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.09 11.709 2 0 0.11 9.128 3 0 0.20 5.032 4 0 0.26 3.779 5 0 0.32 3.158 6 0 0.38 2.635 7 0 0.45 2.243 8 0 0.50 1.989

On 01/31/2008 02:55 PM, William Gropp wrote:

...

The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated.

Bill

On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:

...

Bill,

The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.

-- Pavan

On 01/31/2008 01:17 PM, William Gropp wrote:

...

No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency. Bill On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:

...

Bill,

...

We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.

Do you mean acquire the global lock in the nest count increment?

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

These are very interesting. Is there a chance that the large latency on different cores is because of some occasional scheduling jitter - if one thread is occasionally not running when the others start, there will be one huge latency which will get averaged out in the test. That's less likely if they're all on one core. To test the other issue wrt shared data, can you put together a program that (a) reads from some shared data (like the MPIR_Process structure) and (b) updates some shared data (just a counter, e.g.,), and run a similar test? Other than access the MPIR_Process and PerThread structures, there shouldn't be much shared access in the MPI_PROC_NULL case (if this has the nesting count stuff enabled, that's another test - using pthread_get_specific (or whatever it is called) in the two cases. Bill On Feb 1, 2008, at 12:47 AM, Pavan Balaji wrote:

Ok, we can't do this kind of scheduling on the BG/P (can't have more than one process on a processor). So, I don't have those numbers.

Given this new insight on the impact of process-to-core mappings, we'll need to take step back on what exactly our approach should be. It looks like the problem is not the locks alone. It is probably shared variables that are thrashing in cache. Does this mean we have to rethink how we allocate shared buffers and other objects shared between the different threads?

-- Pavan

On 01/31/2008 09:00 PM, Pavan Balaji wrote:

...

Holy googly moogly! The OS schedules each process on a different core. I forced it to schedule everything on one core and the performance suddenly shot up. Here are the numbers: /* Round robin scheduling of processes to cores */ #bb09# /home/balaji> mpiexec -n 1 ./threads -t 4 -l 100000 -r # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.13 7.899 2 0 1.12 1.780 3 0 3.14 0.955 4 0 4.59 0.871 /* Scheduling all processes on one core */ #bb09# /home/balaji> mpiexec -n 1 ./threads -t 4 -l 100000 # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.13 7.855 2 0 0.26 7.819 3 0 0.38 7.826 4 0 0.51 7.768 -- Pavan On 01/31/2008 07:17 PM, William Gropp wrote:

...

At what point does the OS decide to schedule these threads on different cores? In the classic multithreaded GUI application, the best strategy is to schedule all threads on the same core. Bill

On Jan 31, 2008, at 6:58 PM, Darius Buntinas wrote:

...

Sure, it's on bblogin ~buntinas/ng-mpich/mrate/test/ threads-2.c . It's based on your threads.c that I grabbed from your directory around 4:00.

I moved the pthread_barriers outside the timing loop and ran it again. The "message rate" improved, but still decreased as the number of threads increased.

-d

On 01/31/2008 06:24 PM, Pavan Balaji wrote:

...

Very weird. Can you send me your test program? I'd like to poke into it a bit. -- Pavan On 01/31/2008 06:21 PM, Darius Buntinas wrote:

...

I removed the send and receive calls from the timing loop and replaced them with statements that increment and decrement a volatile variable. The volatile variable is in an array, so each thread accesses its own variable. The array elements are also padded so each is in its own cache line. So there should be no contention between threads on the variable.

It looks like the test program is affecting the results. :-)

# MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 306.547 2 0 0.00 483.213 3 0 0.01 184.908 4 0 0.01 114.825 5 0 0.01 78.911 6 0 0.02 59.759 7 0 0.02 47.333 8 0 0.03 38.067

-d

On 01/31/2008 06:02 PM, Darius Buntinas wrote:

...

I removed the locks and nesting code (i.e., defined CS_ENTER/ EXIT to nothing). I then changed the test to use MPI_Send and MPI_Recv (instead of isend and irecv), so I wouldn't have to use a lock when allocating a request. I then ran it on intel-loaner1 (dual quadcore) sending and receiving to MPI_PROC_NULL. Below are the results (first with locking intact, then with the locking removed).

There's an improvement in message rate, but we still see message rate decrease as we add additional threads. What's interesting, is that even the single threaded case (that doesn't call MPI_Init_thread) shows improvement with locks removed. I can't explain that, since it's not using locks (or at least shouldn't). I'll look into this further.

-d

CONTROL CASE WITH LOCKING INTACT: # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.15 6.774 2 0 0.57 1.755 3 0 1.29 0.773 4 0 2.07 0.483 5 0 2.73 0.366 6 0 3.12 0.321 7 0 3.45 0.290 8 0 3.69 0.271

LOCKING REMOVED: % ./threads-2 -t 8 -l 10000 -pn # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.09 11.709 2 0 0.11 9.128 3 0 0.20 5.032 4 0 0.26 3.779 5 0 0.32 3.158 6 0 0.38 2.635 7 0 0.45 2.243 8 0 0.50 1.989

On 01/31/2008 02:55 PM, William Gropp wrote:

...

The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated.

Bill

On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:

...

Bill,

The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.

-- Pavan

On 01/31/2008 01:17 PM, William Gropp wrote:

...

No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency. Bill On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:

...

Bill,

...

We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.

Do you mean acquire the global lock in the nest count increment?

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

There's always some other processes running (demons etc.), so you do need to be careful about what the OS is doing. The code was designed to minimize the updates of shared structures, so it is odd that the threads on separate cores are showing such bad performance. On the perthread case, I don't know if any special effort was made to ensure that the perthread data structures were on separate cache lines. The accesses to the MPIR_Process structure will also depend on the cache policy on loads - we're hoping that read access is shared, but if it isn't, we'd see something like this. Bill On Feb 1, 2008, at 10:32 AM, Pavan Balaji wrote:

...

Is there a chance that the large latency on different cores is because of some occasional scheduling jitter - if one thread is occasionally not running when the others start, there will be one huge latency which will get averaged out in the test. That's less likely if they're all on one core.

It's unlikely that there's any scheduling jitter. There are exactly the same number of processes as cores in the round-robin case (worse performance case), so there should not be any contention. Also, if there was some jitter, it would show up as noise. The drop in performance with increasing threads is *very* consistent.

There could be scheduling jitter in the single core case since all processes are scheduled on the same core. But this is the case with the higher performance. Also, since the lock anyway serializes the code, it should not matter that the processes are sharing the same core. I agree that this is not the ideal behavior. But it'll let us narrow down the non-lock-related issues.

...

To test the other issue wrt shared data, can you put together a program that (a) reads from some shared data (like the MPIR_Process structure) and (b) updates some shared data (just a counter, e.g.,), and run a similar test?

Ok, I can try this.

...

Other than access the MPIR_Process and PerThread structures, there shouldn't be much shared access in the MPI_PROC_NULL case (if this has the nesting count stuff enabled, that's another test - using pthread_get_specific (or whatever it is called) in the two cases.

Is there a way to find out what data is thrashing across the different cores without analyzing the entire source code? The PerThread stuff should *not* be shared, right? Each thread allocates this individually and access it individually. Also, I think MPIR_Process will only be written to at the start of the program and only read after that. So, that should not cause any cache thrashing either. I think PAPI is my friend here. Let me try that.

Btw, the nesting count stuff was not disabled in the tests I ran. But that's a thread private field too.

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

It looks like the global_mutex within MPIR_ThreadInfo is causing most of the cache misses. But since this is used to get the global lock, we can't really avoid this in a straight forward manner. Anyway, here are some results showing the cache misses too: #bb01# /home/balaji/projects/mpi/threads> ./threads -t 4 # MPI Message Rate Test with 4 threads # nthreads Size Latency (us) Message Rate (MMPS) L2_DCM 1 0 0.35 2.830 17 2 0 0.29 6.875 7 3 0 0.38 7.908 32 4 0 0.51 7.886 29 #bb01# /home/balaji/projects/mpi/threads> ./threads -t 4 -r # MPI Message Rate Test with 4 threads # nthreads Size Latency (us) Message Rate (MMPS) L2_DCM 1 0 0.36 2.785 17 2 0 1.13 1.778 17820628 3 0 2.65 1.134 53253060 4 0 4.20 0.952 93324287 The base case performance (with just one thread) is not the same as what I reported earlier because of some overheads in the FC6 stack. The earlier numbers were on Ubuntu Feisty which was achieving the peak messaging rate for one thread itself. FC6 is taking 3 threads to do the same. But the main focus of these results are the L2 data cache misses. Notice that the cache misses have gone up from nearly zero to tens of millions. Fine-grained locks might improve this overhead as well. I'll work on that next. -- Pavan On 02/01/2008 09:37 AM, William Gropp wrote:

These are very interesting.

Is there a chance that the large latency on different cores is because of some occasional scheduling jitter - if one thread is occasionally not running when the others start, there will be one huge latency which will get averaged out in the test. That's less likely if they're all on one core.

To test the other issue wrt shared data, can you put together a program that (a) reads from some shared data (like the MPIR_Process structure) and (b) updates some shared data (just a counter, e.g.,), and run a similar test? Other than access the MPIR_Process and PerThread structures, there shouldn't be much shared access in the MPI_PROC_NULL case (if this has the nesting count stuff enabled, that's another test - using pthread_get_specific (or whatever it is called) in the two cases.

Bill

On Feb 1, 2008, at 12:47 AM, Pavan Balaji wrote:

...

Ok, we can't do this kind of scheduling on the BG/P (can't have more than one process on a processor). So, I don't have those numbers.

Given this new insight on the impact of process-to-core mappings, we'll need to take step back on what exactly our approach should be. It looks like the problem is not the locks alone. It is probably shared variables that are thrashing in cache. Does this mean we have to rethink how we allocate shared buffers and other objects shared between the different threads?

-- Pavan

On 01/31/2008 09:00 PM, Pavan Balaji wrote:

...

Holy googly moogly! The OS schedules each process on a different core. I forced it to schedule everything on one core and the performance suddenly shot up. Here are the numbers: /* Round robin scheduling of processes to cores */ #bb09# /home/balaji> mpiexec -n 1 ./threads -t 4 -l 100000 -r # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.13 7.899 2 0 1.12 1.780 3 0 3.14 0.955 4 0 4.59 0.871 /* Scheduling all processes on one core */ #bb09# /home/balaji> mpiexec -n 1 ./threads -t 4 -l 100000 # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.13 7.855 2 0 0.26 7.819 3 0 0.38 7.826 4 0 0.51 7.768 -- Pavan On 01/31/2008 07:17 PM, William Gropp wrote:

...

At what point does the OS decide to schedule these threads on different cores? In the classic multithreaded GUI application, the best strategy is to schedule all threads on the same core. Bill

On Jan 31, 2008, at 6:58 PM, Darius Buntinas wrote:

...

Sure, it's on bblogin ~buntinas/ng-mpich/mrate/test/threads-2.c . It's based on your threads.c that I grabbed from your directory around 4:00.

I moved the pthread_barriers outside the timing loop and ran it again. The "message rate" improved, but still decreased as the number of threads increased.

-d

On 01/31/2008 06:24 PM, Pavan Balaji wrote:

...

Very weird. Can you send me your test program? I'd like to poke into it a bit. -- Pavan On 01/31/2008 06:21 PM, Darius Buntinas wrote:

...

I removed the send and receive calls from the timing loop and replaced them with statements that increment and decrement a volatile variable. The volatile variable is in an array, so each thread accesses its own variable. The array elements are also padded so each is in its own cache line. So there should be no contention between threads on the variable.

It looks like the test program is affecting the results. :-)

# MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 306.547 2 0 0.00 483.213 3 0 0.01 184.908 4 0 0.01 114.825 5 0 0.01 78.911 6 0 0.02 59.759 7 0 0.02 47.333 8 0 0.03 38.067

-d

On 01/31/2008 06:02 PM, Darius Buntinas wrote:

...

I removed the locks and nesting code (i.e., defined CS_ENTER/EXIT to nothing). I then changed the test to use MPI_Send and MPI_Recv (instead of isend and irecv), so I wouldn't have to use a lock when allocating a request. I then ran it on intel-loaner1 (dual quadcore) sending and receiving to MPI_PROC_NULL. Below are the results (first with locking intact, then with the locking removed).

There's an improvement in message rate, but we still see message rate decrease as we add additional threads. What's interesting, is that even the single threaded case (that doesn't call MPI_Init_thread) shows improvement with locks removed. I can't explain that, since it's not using locks (or at least shouldn't). I'll look into this further.

-d

CONTROL CASE WITH LOCKING INTACT: # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.15 6.774 2 0 0.57 1.755 3 0 1.29 0.773 4 0 2.07 0.483 5 0 2.73 0.366 6 0 3.12 0.321 7 0 3.45 0.290 8 0 3.69 0.271

LOCKING REMOVED: % ./threads-2 -t 8 -l 10000 -pn # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.09 11.709 2 0 0.11 9.128 3 0 0.20 5.032 4 0 0.26 3.779 5 0 0.32 3.158 6 0 0.38 2.635 7 0 0.45 2.243 8 0 0.50 1.989

On 01/31/2008 02:55 PM, William Gropp wrote:

...

The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated.

Bill

On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:

...

Bill,

The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.

-- Pavan

On 01/31/2008 01:17 PM, William Gropp wrote:

...

No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency. Bill On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:

...

Bill,

...

We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.

Do you mean acquire the global lock in the nest count increment?

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

I agree with Darius's suggestions - an easy test using proc_null is to remove the global mutex from the cs_enter/exit to see if that's the dominant source of the cost. (I'm also not exactly sure how the - r changes the test). With the global mutex in place, if the threads that don't have the lock are busy waiting on it, that traffic might look like a L2 miss. With the global lock in place, at any time, all but one thread will be trying to acquire the lock, I'd expect a lot of memory traffic. The fine grain locks should eliminate that contention. As Darius notes, it would be interesting to know what other sources of L2 misses there might be (e.g., reading from the shared MPIR_Process structure). Bill On Feb 3, 2008, at 9:17 AM, Darius Buntinas wrote:

Are the majority of the cache misses due to the global lock (i.e., MPIU_THREAD_SINGLE_CS_ENTER/_EXIT)? Try commenting that macro out then run the test with blocking communication and proc_null (to avoid the need for a lock).

Another thing to check is to instrument MPICH2 to read only the L2 cache misses around the lock/unlock operations, then read the total L2 cache misses for the MPI calls. I'm just interested in seeing how bad the rest of the code is wrt L2 cache misses.

I'm out of town next week, but I'll try to work on installing PAPI on the intel-loaner machines and running some of these tests.

-d

On 02/03/2008 04:05 AM, Pavan Balaji wrote:

...

It looks like the global_mutex within MPIR_ThreadInfo is causing most of the cache misses. But since this is used to get the global lock, we can't really avoid this in a straight forward manner. Anyway, here are some results showing the cache misses too: #bb01# /home/balaji/projects/mpi/threads> ./threads -t 4 # MPI Message Rate Test with 4 threads # nthreads Size Latency (us) Message Rate (MMPS) L2_DCM 1 0 0.35 2.830 17 2 0 0.29 6.875 7 3 0 0.38 7.908 32 4 0 0.51 7.886 29 #bb01# /home/balaji/projects/mpi/threads> ./threads -t 4 -r # MPI Message Rate Test with 4 threads # nthreads Size Latency (us) Message Rate (MMPS) L2_DCM 1 0 0.36 2.785 17 2 0 1.13 1.778 17820628 3 0 2.65 1.134 53253060 4 0 4.20 0.952 93324287 The base case performance (with just one thread) is not the same as what I reported earlier because of some overheads in the FC6 stack. The earlier numbers were on Ubuntu Feisty which was achieving the peak messaging rate for one thread itself. FC6 is taking 3 threads to do the same. But the main focus of these results are the L2 data cache misses. Notice that the cache misses have gone up from nearly zero to tens of millions. Fine-grained locks might improve this overhead as well. I'll work on that next. -- Pavan On 02/01/2008 09:37 AM, William Gropp wrote:

...

These are very interesting. Is there a chance that the large latency on different cores is because of some occasional scheduling jitter - if one thread is occasionally not running when the others start, there will be one huge latency which will get averaged out in the test. That's less likely if they're all on one core.

To test the other issue wrt shared data, can you put together a program that (a) reads from some shared data (like the MPIR_Process structure) and (b) updates some shared data (just a counter, e.g.,), and run a similar test? Other than access the MPIR_Process and PerThread structures, there shouldn't be much shared access in the MPI_PROC_NULL case (if this has the nesting count stuff enabled, that's another test - using pthread_get_specific (or whatever it is called) in the two cases.

Bill

On Feb 1, 2008, at 12:47 AM, Pavan Balaji wrote:

...

Ok, we can't do this kind of scheduling on the BG/P (can't have more than one process on a processor). So, I don't have those numbers.

Given this new insight on the impact of process-to-core mappings, we'll need to take step back on what exactly our approach should be. It looks like the problem is not the locks alone. It is probably shared variables that are thrashing in cache. Does this mean we have to rethink how we allocate shared buffers and other objects shared between the different threads?

-- Pavan

On 01/31/2008 09:00 PM, Pavan Balaji wrote:

...

Holy googly moogly! The OS schedules each process on a different core. I forced it to schedule everything on one core and the performance suddenly shot up. Here are the numbers: /* Round robin scheduling of processes to cores */ #bb09# /home/balaji> mpiexec -n 1 ./threads -t 4 -l 100000 -r # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.13 7.899 2 0 1.12 1.780 3 0 3.14 0.955 4 0 4.59 0.871 /* Scheduling all processes on one core */ #bb09# /home/balaji> mpiexec -n 1 ./threads -t 4 -l 100000 # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.13 7.855 2 0 0.26 7.819 3 0 0.38 7.826 4 0 0.51 7.768 -- Pavan On 01/31/2008 07:17 PM, William Gropp wrote:

...

At what point does the OS decide to schedule these threads on different cores? In the classic multithreaded GUI application, the best strategy is to schedule all threads on the same core. Bill

On Jan 31, 2008, at 6:58 PM, Darius Buntinas wrote:

...

Sure, it's on bblogin ~buntinas/ng-mpich/mrate/test/ threads-2.c . It's based on your threads.c that I grabbed from your directory around 4:00.

I moved the pthread_barriers outside the timing loop and ran it again. The "message rate" improved, but still decreased as the number of threads increased.

-d

On 01/31/2008 06:24 PM, Pavan Balaji wrote:

...

Very weird. Can you send me your test program? I'd like to poke into it a bit. -- Pavan On 01/31/2008 06:21 PM, Darius Buntinas wrote:

...

I removed the send and receive calls from the timing loop and replaced them with statements that increment and decrement a volatile variable. The volatile variable is in an array, so each thread accesses its own variable. The array elements are also padded so each is in its own cache line. So there should be no contention between threads on the variable.

It looks like the test program is affecting the results. :-)

# MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 306.547 2 0 0.00 483.213 3 0 0.01 184.908 4 0 0.01 114.825 5 0 0.01 78.911 6 0 0.02 59.759 7 0 0.02 47.333 8 0 0.03 38.067

-d

On 01/31/2008 06:02 PM, Darius Buntinas wrote:

...

I removed the locks and nesting code (i.e., defined CS_ENTER/EXIT to nothing). I then changed the test to use MPI_Send and MPI_Recv (instead of isend and irecv), so I wouldn't have to use a lock when allocating a request. I then ran it on intel-loaner1 (dual quadcore) sending and receiving to MPI_PROC_NULL. Below are the results (first with locking intact, then with the locking removed).

There's an improvement in message rate, but we still see message rate decrease as we add additional threads. What's interesting, is that even the single threaded case (that doesn't call MPI_Init_thread) shows improvement with locks removed. I can't explain that, since it's not using locks (or at least shouldn't). I'll look into this further.

-d

CONTROL CASE WITH LOCKING INTACT: # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.15 6.774 2 0 0.57 1.755 3 0 1.29 0.773 4 0 2.07 0.483 5 0 2.73 0.366 6 0 3.12 0.321 7 0 3.45 0.290 8 0 3.69 0.271

LOCKING REMOVED: % ./threads-2 -t 8 -l 10000 -pn # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.09 11.709 2 0 0.11 9.128 3 0 0.20 5.032 4 0 0.26 3.779 5 0 0.32 3.158 6 0 0.38 2.635 7 0 0.45 2.243 8 0 0.50 1.989

On 01/31/2008 02:55 PM, William Gropp wrote:

...

The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated.

Bill

On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:

...

Bill,

The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.

-- Pavan

On 01/31/2008 01:17 PM, William Gropp wrote:

...

No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency. Bill On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:

...

Bill,

...

We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.

Do you mean acquire the global lock in the nest count increment?

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

Btw, the code has grown out of a single file. I've added it into a temporary SVN repository. You can check it out using: $ svn co svn+ssh://shakey/sandbox/balaji/svn/projects/threads/trunk threads Thanks. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

Bill,

The results make sense if thread_lock polls on a memory location - when the threads are on different cores, they (may) have to take the L2 misses.

I verified online that AMD Opterons do have a shared state for their caches (they use MOESI coherency). So, we should not see L2 cache misses till someone updates the lock, i.e., either locks or unlocks. Empty polls should just change the cache to the shared state. Of course, it is possible (and likely?) that the lock keeps getting acquired and released if all threads call blocking calls, since when a thread blocks it releases the lock and tries to get it back once it receives an event.

If the thread_lock is waiting for an event, the cost of a lock will be very large (compared to a cache miss), as an event manager will be involved (such as the OS). So I hope that the mutex is implemented to poll (and possibly yield); that's what you'd want for threads that should run concurrently. I'd guess that the thread yields after a few tries; that would let the other threads on the same core run efficiently.

This seems logical. However, I wrote a simple program with two threads, where the first thread acquires the lock and sleeps for 10 seconds, while the second thread waits for the lock to get released. In this case, if the wait was polling, I should notice 100% usage on one core. But, I don't see that. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

Darius, I ran a bunch of tests last night. I didn't send out the details in the previous email, as they were too many cases. Here are the details for the ones you asked for:

Are the majority of the cache misses due to the global lock (i.e., MPIU_THREAD_SINGLE_CS_ENTER/_EXIT)? Try commenting that macro out then run the test with blocking communication and proc_null (to avoid the need for a lock).

Note that the blocking send call also calls the lock. The lock is released only when the call actually blocks for an event. So there are some cache misses even in this case, though lesser than in the non-blocking version. I also tried getting rid of the lock and running (an incorrect version of) the test. But that seg faults.

Another thing to check is to instrument MPICH2 to read only the L2 cache misses around the lock/unlock operations, then read the total L2 cache misses for the MPI calls. I'm just interested in seeing how bad the rest of the code is wrt L2 cache misses.

Yup, tried that too. Well, in the blocking case I got MPI_Send to return immediately after the lock/unlock and there was not much change in the cache misses. So, it looks like the lock/unlock is the major contributor to the cache misses. For the MPI_Isend case, that's more difficult to figure out. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

OK I fixed a bug in the test program that calculated the message rate. I then kept the number of threads created constant and kept the extra ones idle (waiting in a pthread barrier). This should normalize differences in scheduling, barrier times, etc. Now when I run the dummy test (no MPI calls) I get expected results: % ./threads-2 -t 8 -l 10000000 # MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 452.149 2 0 0.00 444.716 3 0 0.00 442.400 4 0 0.00 444.068 5 0 0.00 441.870 6 0 0.00 442.154 7 0 0.00 442.156 8 0 0.00 442.810 Running on a single core shows "message rate" decreasing as the number of active threads increase, as expected. % taskset -c 1 ./threads-2 -t 8 -l 10000000 # MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 454.413 2 0 0.00 226.710 3 0 0.01 150.964 4 0 0.01 113.207 5 0 0.01 90.535 6 0 0.01 75.443 7 0 0.02 64.649 8 0 0.02 56.562 I'll run some non-dummy tests and see what happens. -d On 01/31/2008 06:21 PM, Darius Buntinas wrote:

I removed the send and receive calls from the timing loop and replaced them with statements that increment and decrement a volatile variable. The volatile variable is in an array, so each thread accesses its own variable. The array elements are also padded so each is in its own cache line. So there should be no contention between threads on the variable.

It looks like the test program is affecting the results. :-)

# MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 306.547 2 0 0.00 483.213 3 0 0.01 184.908 4 0 0.01 114.825 5 0 0.01 78.911 6 0 0.02 59.759 7 0 0.02 47.333 8 0 0.03 38.067

-d

On 01/31/2008 06:02 PM, Darius Buntinas wrote:

...

I removed the locks and nesting code (i.e., defined CS_ENTER/EXIT to nothing). I then changed the test to use MPI_Send and MPI_Recv (instead of isend and irecv), so I wouldn't have to use a lock when allocating a request. I then ran it on intel-loaner1 (dual quadcore) sending and receiving to MPI_PROC_NULL. Below are the results (first with locking intact, then with the locking removed).

There's an improvement in message rate, but we still see message rate decrease as we add additional threads. What's interesting, is that even the single threaded case (that doesn't call MPI_Init_thread) shows improvement with locks removed. I can't explain that, since it's not using locks (or at least shouldn't). I'll look into this further.

-d

CONTROL CASE WITH LOCKING INTACT: # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.15 6.774 2 0 0.57 1.755 3 0 1.29 0.773 4 0 2.07 0.483 5 0 2.73 0.366 6 0 3.12 0.321 7 0 3.45 0.290 8 0 3.69 0.271

LOCKING REMOVED: % ./threads-2 -t 8 -l 10000 -pn # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.09 11.709 2 0 0.11 9.128 3 0 0.20 5.032 4 0 0.26 3.779 5 0 0.32 3.158 6 0 0.38 2.635 7 0 0.45 2.243 8 0 0.50 1.989

On 01/31/2008 02:55 PM, William Gropp wrote:

...

The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated.

Bill

On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:

...

Bill,

The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.

-- Pavan

On 01/31/2008 01:17 PM, William Gropp wrote:

...

No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency. Bill On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:

...

Bill,

...

We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.

Do you mean acquire the global lock in the nest count increment?

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

Did you fix the t_start, t_end problem, i.e. make them thread private or have only the main thread measure the time. Anyhow only the main thread is printing the time. Rajeev

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Thursday, January 31, 2008 8:00 AM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

Below are the numbers while using MPI_PROC_NULL. The trend is the same. I guess the next step is to start instrumenting the code.

BG/P: ---- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 1.02 0.981 2 0 1.21 0.825 3 0 2.48 0.403 4 0 3.78 0.265

Breadboard: ---------- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.39 2.589 2 0 0.85 1.183 3 0 4.68 0.213 4 0 4.36 0.229

Thanks.

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

In the code I have, the message rate is calculated as nthreads/latency. However, the numbers below seem to be calculated as 1/latency. Which is right? Rajeev

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Thursday, January 31, 2008 8:00 AM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

Below are the numbers while using MPI_PROC_NULL. The trend is the same. I guess the next step is to start instrumenting the code.

BG/P: ---- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 1.02 0.981 2 0 1.21 0.825 3 0 2.48 0.403 4 0 3.78 0.265

Breadboard: ---------- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.39 2.589 2 0 0.85 1.183 3 0 4.68 0.213 4 0 4.36 0.229

Thanks.

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

No, I think it's taken care of in your new code because you are dividing latency by the number of threads. Rajeev

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Thursday, January 31, 2008 10:44 AM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

Oh, good catch. I think it should be nthreads/latency. My bad.

-- Pavan

On 01/31/2008 10:36 AM, Rajeev Thakur wrote:

...

In the code I have, the message rate is calculated as nthreads/latency. However, the numbers below seem to be calculated as 1/latency. Which is right?

Rajeev

...

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Thursday, January 31, 2008 8:00 AM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

Below are the numbers while using MPI_PROC_NULL. The trend is the same. I guess the next step is to start instrumenting the code.

BG/P: ---- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 1.02 0.981 2 0 1.21 0.825 3 0 2.48 0.403 4 0 3.78 0.265

Breadboard: ---------- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.39 2.589 2 0 0.85 1.183 3 0 4.68 0.213 4 0 4.36 0.229

Thanks.

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

Ah yes.

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Thursday, January 31, 2008 10:48 AM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

Actually that's what I thought when I wrote the code, but the latency number is also being aggregated. That means we have to multiply by nt^2. That is, average latency = total_latency / nt (this is already there). But total messaging rate = nt / average_latency. I'm missing the second nt.

-- Pavan

On 01/31/2008 10:45 AM, Rajeev Thakur wrote:

...

No, I think it's taken care of in your new code because you are dividing latency by the number of threads.

Rajeev

...

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Thursday, January 31, 2008 10:44 AM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

Oh, good catch. I think it should be nthreads/latency. My bad.

-- Pavan

On 01/31/2008 10:36 AM, Rajeev Thakur wrote:

...

In the code I have, the message rate is calculated as nthreads/latency. However, the numbers below seem to be calculated as 1/latency. Which is right?

Rajeev

...

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Thursday, January 31, 2008 8:00 AM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

Below are the numbers while using MPI_PROC_NULL. The trend is the same. I guess the next step is to start instrumenting the code.

BG/P: ---- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 1.02 0.981 2 0 1.21 0.825 3 0 2.48 0.403 4 0 3.78 0.265

Breadboard: ---------- # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.39 2.589 2 0 0.85 1.183 3 0 4.68 0.213 4 0 4.36 0.229

Thanks.

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

Actually, you can do something like the halo test I've added to mpptest to check that a single process can use multiple links concurrently. In that case, you really need a system with 2t cores, where t processes can run 1 thread and 1 process runs t threads; each of those t threads communicates with a uniquely chosen process. That will test sending on t different VCs; you only time the process with t threads. Along the lines of the experiments that Rajeev and I did, users will expect the overhead to be small, so in the current test, the messaging rate should stay relatively constant (actually increasing slightly as the non-communication part can be run concurrently, assuming we avoid the one big lock - that, by the way, is anther test - just use the per VC lock, and disable to global lock. This is also wrong, but does help set a limit on the potential performance with the current code. A harder question is can the lock on the VC also be narrowed so that threads communicating on the same VC spend as little time as possible coordinating access to the VC. Eg, if there was a separate thread performing at actions on the VC, then the only time involved for the compute/user threads would be enqueueing or dequeueing an operation. Bill On Jan 30, 2008, at 1:07 PM, Darius Buntinas wrote:

I just want to check what the benchmark should do.

My understanding from the conference calls and Sameer's slides is that they're interested in the case where multiple threads of a process are sending/receiving to/from different processes. This allows the process to use multiple DMAs concurrently.

Right now, we've been testing the case where one process is sending/ receiving with itself. So moving the locks around VCs may decrease the size of the CS, but will still serialize.

So what's the test we want to run?

I imagine something like P processes each with P-1 threads where each thread is communicating with a different process. But then we need a machine with P(P-1) cores.

On the 8-core nodes we have, we can run 3 processes with 2 threads each. But I think we need more threads to really stress things.

It seems that what we can see from the one or two processes is the just the effect of the size of the CS.

Comments?

-d

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

Yeah, I misunderstood the code. Darius pointed it out too. However, it again comes back to what exactly we want to measure -- threading overhead in MPICH2 or threading overhead in the overall communication and networking stack? -- Pavan On 01/30/2008 01:57 PM, Rajeev Thakur wrote:

Well the original code IBM sent does not have this 1 process limitation. See attached. The threads of a process communicate with the threads of another process. I think that is a better test. Why did we change it?

Rajeev

...

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Darius Buntinas Sent: Wednesday, January 30, 2008 1:07 PM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

I just want to check what the benchmark should do.

My understanding from the conference calls and Sameer's slides is that they're interested in the case where multiple threads of a process are sending/receiving to/from different processes. This allows the process to use multiple DMAs concurrently.

Right now, we've been testing the case where one process is sending/receiving with itself. So moving the locks around VCs may decrease the size of the CS, but will still serialize.

So what's the test we want to run?

I imagine something like P processes each with P-1 threads where each thread is communicating with a different process. But then we need a machine with P(P-1) cores.

On the 8-core nodes we have, we can run 3 processes with 2 threads each. But I think we need more threads to really stress things.

It seems that what we can see from the one or two processes is the just the effect of the size of the CS.

Comments?

-d

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

Because of how MPICH2 is implemented, there's one test that we could use to measure the impact of the global lock. Have t threads send to/receive from MPI_PROC_NULL . The test for this destination is, IIRC, deep in the ch3 code. This will avoid *all* of the communication device while measuring most of the MPICH2 stack. Bill On Jan 30, 2008, at 3:03 PM, Rajeev Thakur wrote:

I would say threading overhead as might be seen by an application. The tests in www.mcs.anl.gov/~thakur/thread-tests also try to measure that. I will point Sameer at those.

Rajeev

...

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Wednesday, January 30, 2008 2:57 PM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

Yeah, I misunderstood the code. Darius pointed it out too. However, it again comes back to what exactly we want to measure -- threading overhead in MPICH2 or threading overhead in the overall communication and networking stack?

-- Pavan

On 01/30/2008 01:57 PM, Rajeev Thakur wrote:

...

Well the original code IBM sent does not have this 1 process limitation. See attached. The threads of a process communicate with the threads of another process. I think that is a better test. Why did we change it?

Rajeev

...

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Darius Buntinas Sent: Wednesday, January 30, 2008 1:07 PM To: [email protected] Subject: Re: [mpich2-core] MPICH2 Threading Overhead

I just want to check what the benchmark should do.

My understanding from the conference calls and Sameer's slides is that they're interested in the case where multiple threads of a process are sending/receiving to/from different processes. This allows the process to use multiple DMAs concurrently.

Right now, we've been testing the case where one process is sending/receiving with itself. So moving the locks around VCs may decrease the size of the CS, but will still serialize.

So what's the test we want to run?

I imagine something like P processes each with P-1 threads where each thread is communicating with a different process. But then we need a machine with P(P-1) cores.

On the 8-core nodes we have, we can run 3 processes with 2 threads each. But I think we need more threads to really stress things.

It seems that what we can see from the one or two processes is the just the effect of the size of the CS.

Comments?

-d

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign