Datatype performance FYI
This is just a heads up - inspired by a paper that Torsten will present at EuroMPI, I created some new datatype performance tests (in Torsten's paper, Open MPI does much better than MPICH2). These are drawn from applications and are cases that users expect to perform well. I've also identified some of the reasons: 1) contiguous blocks aren't merged into the parent datatype (nestvec) 2) struct always flattens types, even when this is not appropriate (nestvec2) 3) block index has high overhead for small (e.g., one word) blocks. (indexperf) These should really be compiled with optimization (not currently done, so you'll need to change the Makefile in perf by hand), where 10x performance problems become apparent. In general, the data loop code, except for the simple vector case, has high overhead and is likely to be slow for small blocks, which are common in applications. The data loop concept permits optimizations for this case (see the vector case), but it looks like those were never done. I have a partial fix for the first case, and its pretty easy to turn on or off the struct flattening. Harder is to flatten structs when it makes sense. Fixing #3 requires doing the same sort of code that was used for vectors; that should also be applied to indexed types. Even better is to generate good code based on the platform; I have a student working on that. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
Can you send us the pdf of the paper? Rajeev On Aug 1, 2012, at 5:06 PM, William Gropp wrote:
This is just a heads up - inspired by a paper that Torsten will present at EuroMPI, I created some new datatype performance tests (in Torsten's paper, Open MPI does much better than MPICH2). These are drawn from applications and are cases that users expect to perform well. I've also identified some of the reasons:
1) contiguous blocks aren't merged into the parent datatype (nestvec) 2) struct always flattens types, even when this is not appropriate (nestvec2) 3) block index has high overhead for small (e.g., one word) blocks. (indexperf)
These should really be compiled with optimization (not currently done, so you'll need to change the Makefile in perf by hand), where 10x performance problems become apparent.
In general, the data loop code, except for the simple vector case, has high overhead and is likely to be slow for small blocks, which are common in applications. The data loop concept permits optimizations for this case (see the vector case), but it looks like those were never done.
I have a partial fix for the first case, and its pretty easy to turn on or off the struct flattening. Harder is to flatten structs when it makes sense. Fixing #3 requires doing the same sort of code that was used for vectors; that should also be applied to indexed types. Even better is to generate good code based on the platform; I have a student working on that.
Bill
William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
I've asked Torsten. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign On Aug 1, 2012, at 5:07 PM, Rajeev Thakur wrote:
Can you send us the pdf of the paper?
Rajeev
Torsten has made it available at http://www.unixer.de/publications/index.php?pub=151 . I also have his test code; I've built some code that helped me identify the form of the generated data loop. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign On Aug 1, 2012, at 5:07 PM, Rajeev Thakur wrote:
Can you send us the pdf of the paper?
Rajeev
There's debugging code in the tree that will dump some of the data structures also, see src/mpid/common/datatype/dataloop/dataloop.c. Rob On Aug 2, 2012, at 9:06 AM, William Gropp wrote:
Torsten has made it available at
http://www.unixer.de/publications/index.php?pub=151 .
I also have his test code; I've built some code that helped me identify the form of the generated data loop.
Bill
William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
On Aug 1, 2012, at 5:07 PM, Rajeev Thakur wrote:
Can you send us the pdf of the paper?
Rajeev
Yep, I found that and it helped; I added some that helped me with some specific questions. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign On Aug 2, 2012, at 11:30 AM, Rob Ross wrote:
There's debugging code in the tree that will dump some of the data structures also, see src/mpid/common/datatype/dataloop/dataloop.c.
Rob
On Aug 2, 2012, at 9:06 AM, William Gropp wrote:
Torsten has made it available at
http://www.unixer.de/publications/index.php?pub=151 .
I also have his test code; I've built some code that helped me identify the form of the generated data loop.
Bill
William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
On Aug 1, 2012, at 5:07 PM, Rajeev Thakur wrote:
Can you send us the pdf of the paper?
Rajeev
We never implemented optimizations that required combining multiple dataloops into one. This sounds like (1) and (3)? -- Rob On Aug 1, 2012, at 5:06 PM, William Gropp wrote:
This is just a heads up - inspired by a paper that Torsten will present at EuroMPI, I created some new datatype performance tests (in Torsten's paper, Open MPI does much better than MPICH2). These are drawn from applications and are cases that users expect to perform well. I've also identified some of the reasons:
1) contiguous blocks aren't merged into the parent datatype (nestvec) 2) struct always flattens types, even when this is not appropriate (nestvec2) 3) block index has high overhead for small (e.g., one word) blocks. (indexperf)
These should really be compiled with optimization (not currently done, so you'll need to change the Makefile in perf by hand), where 10x performance problems become apparent.
In general, the data loop code, except for the simple vector case, has high overhead and is likely to be slow for small blocks, which are common in applications. The data loop concept permits optimizations for this case (see the vector case), but it looks like those were never done.
I have a partial fix for the first case, and its pretty easy to turn on or off the struct flattening. Harder is to flatten structs when it makes sense. Fixing #3 requires doing the same sort of code that was used for vectors; that should also be applied to indexed types. Even better is to generate good code based on the platform; I have a student working on that.
Bill
William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
Its #1 (this is the simplest combination). #3 is different. The code for packing from block indexed for single word data should look roughly like while (count--) *outbuf++ = inbuf[*id++]; with the appropriate variations for size, aligned types, size of outbuf (but all such checks hoisted out of the loop). With const and restrict, a really good compiler can generate good code for this; for other compilers, the usual tricks can be applied. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign On Aug 1, 2012, at 5:17 PM, Rob Ross wrote:
We never implemented optimizations that required combining multiple dataloops into one. This sounds like (1) and (3)? -- Rob
On Aug 1, 2012, at 5:06 PM, William Gropp wrote:
This is just a heads up - inspired by a paper that Torsten will present at EuroMPI, I created some new datatype performance tests (in Torsten's paper, Open MPI does much better than MPICH2). These are drawn from applications and are cases that users expect to perform well. I've also identified some of the reasons:
1) contiguous blocks aren't merged into the parent datatype (nestvec) 2) struct always flattens types, even when this is not appropriate (nestvec2) 3) block index has high overhead for small (e.g., one word) blocks. (indexperf)
These should really be compiled with optimization (not currently done, so you'll need to change the Makefile in perf by hand), where 10x performance problems become apparent.
In general, the data loop code, except for the simple vector case, has high overhead and is likely to be slow for small blocks, which are common in applications. The data loop concept permits optimizations for this case (see the vector case), but it looks like those were never done.
I have a partial fix for the first case, and its pretty easy to turn on or off the struct flattening. Harder is to flatten structs when it makes sense. Fixing #3 requires doing the same sort of code that was used for vectors; that should also be applied to indexed types. Even better is to generate good code based on the platform; I have a student working on that.
Bill
William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
I see. Sounds like #3 can be handled by optimization in the leaf function, then? That one should be easy. -- Rob On Aug 2, 2012, at 9:01 AM, William Gropp wrote:
Its #1 (this is the simplest combination). #3 is different. The code for packing from block indexed for single word data should look roughly like
while (count--) *outbuf++ = inbuf[*id++];
with the appropriate variations for size, aligned types, size of outbuf (but all such checks hoisted out of the loop). With const and restrict, a really good compiler can generate good code for this; for other compilers, the usual tricks can be applied.
Bill
William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
On Aug 1, 2012, at 5:17 PM, Rob Ross wrote:
We never implemented optimizations that required combining multiple dataloops into one. This sounds like (1) and (3)? -- Rob
On Aug 1, 2012, at 5:06 PM, William Gropp wrote:
This is just a heads up - inspired by a paper that Torsten will present at EuroMPI, I created some new datatype performance tests (in Torsten's paper, Open MPI does much better than MPICH2). These are drawn from applications and are cases that users expect to perform well. I've also identified some of the reasons:
1) contiguous blocks aren't merged into the parent datatype (nestvec) 2) struct always flattens types, even when this is not appropriate (nestvec2) 3) block index has high overhead for small (e.g., one word) blocks. (indexperf)
These should really be compiled with optimization (not currently done, so you'll need to change the Makefile in perf by hand), where 10x performance problems become apparent.
In general, the data loop code, except for the simple vector case, has high overhead and is likely to be slow for small blocks, which are common in applications. The data loop concept permits optimizations for this case (see the vector case), but it looks like those were never done.
I have a partial fix for the first case, and its pretty easy to turn on or off the struct flattening. Harder is to flatten structs when it makes sense. Fixing #3 requires doing the same sort of code that was used for vectors; that should also be applied to indexed types. Even better is to generate good code based on the platform; I have a student working on that.
Bill
William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
Yes, it looks like the kind of optimizations applied to vectors need to be applied to the other basic data loops. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign On Aug 2, 2012, at 11:27 AM, Rob Ross wrote:
I see. Sounds like #3 can be handled by optimization in the leaf function, then? That one should be easy. -- Rob
On Aug 2, 2012, at 9:01 AM, William Gropp wrote:
Its #1 (this is the simplest combination). #3 is different. The code for packing from block indexed for single word data should look roughly like
while (count--) *outbuf++ = inbuf[*id++];
with the appropriate variations for size, aligned types, size of outbuf (but all such checks hoisted out of the loop). With const and restrict, a really good compiler can generate good code for this; for other compilers, the usual tricks can be applied.
Bill
William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
On Aug 1, 2012, at 5:17 PM, Rob Ross wrote:
We never implemented optimizations that required combining multiple dataloops into one. This sounds like (1) and (3)? -- Rob
On Aug 1, 2012, at 5:06 PM, William Gropp wrote:
This is just a heads up - inspired by a paper that Torsten will present at EuroMPI, I created some new datatype performance tests (in Torsten's paper, Open MPI does much better than MPICH2). These are drawn from applications and are cases that users expect to perform well. I've also identified some of the reasons:
1) contiguous blocks aren't merged into the parent datatype (nestvec) 2) struct always flattens types, even when this is not appropriate (nestvec2) 3) block index has high overhead for small (e.g., one word) blocks. (indexperf)
These should really be compiled with optimization (not currently done, so you'll need to change the Makefile in perf by hand), where 10x performance problems become apparent.
In general, the data loop code, except for the simple vector case, has high overhead and is likely to be slow for small blocks, which are common in applications. The data loop concept permits optimizations for this case (see the vector case), but it looks like those were never done.
I have a partial fix for the first case, and its pretty easy to turn on or off the struct flattening. Harder is to flatten structs when it makes sense. Fixing #3 requires doing the same sort of code that was used for vectors; that should also be applied to indexed types. Even better is to generate good code based on the platform; I have a student working on that.
Bill
William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
participants (3)
-
Rajeev Thakur -
Rob Ross -
William Gropp