The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
Hi Bill, We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests). Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run. -- Pavan On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory).
Bill
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority. -Dave On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping. Bill On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote:
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority.
-Dave
On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable-nemesis-dbg-localoddeven. Rajeev _____ From: [email protected] [mailto:[email protected]] On Behalf Of William Gropp Sent: Friday, July 25, 2008 11:31 AM To: [email protected] Subject: Re: [mpich2-core] Question about tests Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping. Bill On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote: The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority. -Dave On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote: Hi Bill, We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests). Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run. -- Pavan On 07/25/2008 09:12 AM, William Gropp wrote: The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign -- Pavan Balaji http://www.mcs.anl.gov/~balaji William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
Sure, though it will be a hack. Is there some way to force nemesis to use TCP instead of shared memory for all connections? Bill On Jul 25, 2008, at 12:02 PM, Rajeev Thakur wrote:
Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable- nemesis-dbg-localoddeven.
Rajeev
From: [email protected] [mailto:owner-mpich2- [email protected]] On Behalf Of William Gropp Sent: Friday, July 25, 2008 11:31 AM To: [email protected] Subject: Re: [mpich2-core] Question about tests
Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping.
Bill
On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote:
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority.
-Dave
On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
--enable-nemesis-dbg-nolocal -d On 07/28/2008 11:32 AM, William Gropp wrote:
Sure, though it will be a hack.
Is there some way to force nemesis to use TCP instead of shared memory for all connections?
Bill
On Jul 25, 2008, at 12:02 PM, Rajeev Thakur wrote:
Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable-nemesis-dbg-localoddeven.
Rajeev
------------------------------------------------------------------------ *From:* [email protected] [mailto:[email protected]] *On Behalf Of *William Gropp *Sent:* Friday, July 25, 2008 11:31 AM *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [mpich2-core] Question about tests
Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping.
Bill
On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote:
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority.
-Dave
On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
Do we want to do this or the TCP + shared memory option? Rajeev
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Darius Buntinas Sent: Monday, July 28, 2008 11:37 AM To: [email protected] Subject: Re: [mpich2-core] Question about tests
--enable-nemesis-dbg-nolocal -d
On 07/28/2008 11:32 AM, William Gropp wrote:
Sure, though it will be a hack.
Is there some way to force nemesis to use TCP instead of shared memory for all connections?
Bill
On Jul 25, 2008, at 12:02 PM, Rajeev Thakur wrote:
Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable-nemesis-dbg-localoddeven.
Rajeev
----------
*From:* [email protected] [mailto:[email protected]] *On Behalf Of
*William Gropp
*Sent:* Friday, July 25, 2008 11:31 AM *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [mpich2-core] Question about tests
Thanks for the info. I'll still stand by the "if its
not tested it
doesn't work" position, so it would be good to do at
least a basic
test (by hand if that's easier) before shipping.
Bill
On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote:
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain
kinds of bugs
in the new(ish) MPIDI_GetIPInterfaces function that
they might not be
detected on only one node. We might need to set up a
nightly joint
test between two of the non-breadboard machines if we
do want to be
sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority.
-Dave
On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option
which means
that all odd ranks form a shared-memory sub-cluster
all even ranks
do the same. Communication between an odd and an
even rank process
happens through the network. So, both should be
getting tested.
Real two-node testing is better, but we were trying
to optimize for
high-throughput computing given that we have too few
machines and
too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in
conjunction so
that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started
included some tests
between different machines to ensure that
node-to-node worked.
The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls
"Results from
the MPICH2 test suite". (My concern is that Nemesis
is detecting
that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never
shared memory).
Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
Ergh, is there a way to do this at runtime? Could there be? Bill On Jul 28, 2008, at 11:37 AM, Darius Buntinas wrote:
--enable-nemesis-dbg-nolocal -d
On 07/28/2008 11:32 AM, William Gropp wrote:
Sure, though it will be a hack.
Is there some way to force nemesis to use TCP instead of shared memory for all connections?
Bill
On Jul 25, 2008, at 12:02 PM, Rajeev Thakur wrote:
Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable-nemesis-dbg-localoddeven.
Rajeev
-------------------------------------------------------------------- ---- *From:* [email protected] [mailto:[email protected]] *On Behalf Of *William Gropp *Sent:* Friday, July 25, 2008 11:31 AM *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [mpich2-core] Question about tests
Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping.
Bill
On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote:
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority.
-Dave
On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
I think you can set MPICH_NO_LOCAL=1 or MPICH_ODD_EVEN_CLIQUES=1, although I haven't used it myself so I'm not sure if it works. -Dave On Jul 28, 2008, at 11:42 AM, William Gropp wrote:
Ergh, is there a way to do this at runtime? Could there be?
Bill
On Jul 28, 2008, at 11:37 AM, Darius Buntinas wrote:
--enable-nemesis-dbg-nolocal -d
On 07/28/2008 11:32 AM, William Gropp wrote:
Sure, though it will be a hack.
Is there some way to force nemesis to use TCP instead of shared memory for all connections?
Bill
On Jul 25, 2008, at 12:02 PM, Rajeev Thakur wrote:
Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable-nemesis-dbg-localoddeven.
Rajeev
------------------------------------------------------------------- ----- *From:* [email protected] [mailto:[email protected]] *On Behalf Of *William Gropp *Sent:* Friday, July 25, 2008 11:31 AM *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [mpich2-core] Question about tests
Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping.
Bill
On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote:
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority.
-Dave
On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
Sure, MPICH_NO_LOCAL or MPICH_ODD_EVEN_CLIQUES environment variables. -d On 07/28/2008 11:42 AM, William Gropp wrote:
Ergh, is there a way to do this at runtime? Could there be?
Bill
On Jul 28, 2008, at 11:37 AM, Darius Buntinas wrote:
--enable-nemesis-dbg-nolocal -d
On 07/28/2008 11:32 AM, William Gropp wrote:
Sure, though it will be a hack.
Is there some way to force nemesis to use TCP instead of shared memory for all connections?
Bill
On Jul 25, 2008, at 12:02 PM, Rajeev Thakur wrote:
Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable-nemesis-dbg-localoddeven.
Rajeev
------------------------------------------------------------------------ *From:* [email protected] <mailto:[email protected]> [mailto:[email protected]] *On Behalf Of *William Gropp *Sent:* Friday, July 25, 2008 11:31 AM *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [mpich2-core] Question about tests
Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping.
Bill
On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote:
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority.
-Dave
On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
Is there a way to test that setting MPICH_NO_LOCAL=1 works? For example, are there DBG hooks so that I can use the debug features to print out the choice of method for vc's? Bill On Jul 28, 2008, at 12:46 PM, Darius Buntinas wrote:
Sure, MPICH_NO_LOCAL or MPICH_ODD_EVEN_CLIQUES environment variables.
-d
On 07/28/2008 11:42 AM, William Gropp wrote:
Ergh, is there a way to do this at runtime? Could there be?
Bill
On Jul 28, 2008, at 11:37 AM, Darius Buntinas wrote:
--enable-nemesis-dbg-nolocal -d
On 07/28/2008 11:32 AM, William Gropp wrote:
Sure, though it will be a hack.
Is there some way to force nemesis to use TCP instead of shared memory for all connections?
Bill
On Jul 25, 2008, at 12:02 PM, Rajeev Thakur wrote:
Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable-nemesis-dbg-localoddeven.
Rajeev
------------------------------------------------------------------ ------ *From:* [email protected] <mailto:[email protected]> [mailto:[email protected]] *On Behalf Of *William Gropp *Sent:* Friday, July 25, 2008 11:31 AM *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [mpich2-core] Question about tests
Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping.
Bill
On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote:
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority.
-Dave
On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
An immediate one is that the RMA tests should fail. Rajeev _____ From: [email protected] [mailto:[email protected]] On Behalf Of William Gropp Sent: Monday, July 28, 2008 5:58 PM To: [email protected] Subject: Re: [mpich2-core] Question about tests Is there a way to test that setting MPICH_NO_LOCAL=1 works? For example, are there DBG hooks so that I can use the debug features to print out the choice of method for vc's? Bill On Jul 28, 2008, at 12:46 PM, Darius Buntinas wrote: Sure, MPICH_NO_LOCAL or MPICH_ODD_EVEN_CLIQUES environment variables. -d On 07/28/2008 11:42 AM, William Gropp wrote: Ergh, is there a way to do this at runtime? Could there be? Bill On Jul 28, 2008, at 11:37 AM, Darius Buntinas wrote: --enable-nemesis-dbg-nolocal -d On 07/28/2008 11:32 AM, William Gropp wrote: Sure, though it will be a hack. Is there some way to force nemesis to use TCP instead of shared memory for all connections? Bill On Jul 25, 2008, at 12:02 PM, Rajeev Thakur wrote: Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable-nemesis-dbg-localoddeven. Rajeev ------------------------------------------------------------------------ *From:* [email protected] <mailto:[email protected]> [mailto:[email protected]] *On Behalf Of *William Gropp *Sent:* Friday, July 25, 2008 11:31 AM *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [mpich2-core] Question about tests Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping. Bill On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote: The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority. -Dave On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote: Hi Bill, We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests). Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run. -- Pavan On 07/25/2008 09:12 AM, William Gropp wrote: The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign -- Pavan Balaji http://www.mcs.anl.gov/~balaji William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
I'll take a look for debugging messages, but you should be able to tell with a latency test. -d On 07/28/2008 05:58 PM, William Gropp wrote:
Is there a way to test that setting MPICH_NO_LOCAL=1 works? For example, are there DBG hooks so that I can use the debug features to print out the choice of method for vc's?
Bill
On Jul 28, 2008, at 12:46 PM, Darius Buntinas wrote:
Sure, MPICH_NO_LOCAL or MPICH_ODD_EVEN_CLIQUES environment variables.
-d
On 07/28/2008 11:42 AM, William Gropp wrote:
Ergh, is there a way to do this at runtime? Could there be?
Bill
On Jul 28, 2008, at 11:37 AM, Darius Buntinas wrote:
--enable-nemesis-dbg-nolocal -d
On 07/28/2008 11:32 AM, William Gropp wrote:
Sure, though it will be a hack.
Is there some way to force nemesis to use TCP instead of shared memory for all connections?
Bill
On Jul 25, 2008, at 12:02 PM, Rajeev Thakur wrote:
Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable-nemesis-dbg-localoddeven.
Rajeev
------------------------------------------------------------------------ *From:* [email protected] <mailto:[email protected]> <mailto:[email protected]> [mailto:[email protected]] *On Behalf Of *William Gropp *Sent:* Friday, July 25, 2008 11:31 AM *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [mpich2-core] Question about tests
Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping.
Bill
On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote:
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority.
-Dave
On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
No I don't see any dbg log messages there. If you tell me exactly what you're looking for, I can add some. -d On 07/28/2008 05:58 PM, William Gropp wrote:
Is there a way to test that setting MPICH_NO_LOCAL=1 works? For example, are there DBG hooks so that I can use the debug features to print out the choice of method for vc's?
Bill
On Jul 28, 2008, at 12:46 PM, Darius Buntinas wrote:
Sure, MPICH_NO_LOCAL or MPICH_ODD_EVEN_CLIQUES environment variables.
-d
On 07/28/2008 11:42 AM, William Gropp wrote:
Ergh, is there a way to do this at runtime? Could there be?
Bill
On Jul 28, 2008, at 11:37 AM, Darius Buntinas wrote:
--enable-nemesis-dbg-nolocal -d
On 07/28/2008 11:32 AM, William Gropp wrote:
Sure, though it will be a hack.
Is there some way to force nemesis to use TCP instead of shared memory for all connections?
Bill
On Jul 25, 2008, at 12:02 PM, Rajeev Thakur wrote:
Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable-nemesis-dbg-localoddeven.
Rajeev
------------------------------------------------------------------------ *From:* [email protected] <mailto:[email protected]> <mailto:[email protected]> [mailto:[email protected]] *On Behalf Of *William Gropp *Sent:* Friday, July 25, 2008 11:31 AM *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [mpich2-core] Question about tests
Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping.
Bill
On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote:
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority.
-Dave
On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
I'd like the dbg option to tell me when a particular netmod is selected for a vc - the class could be MPIU_DBG_VC or MPIU_DBG_COMM_METHOD (a new class, defined since other devices may want to report on the same thing). Bill On Jul 29, 2008, at 10:07 AM, Darius Buntinas wrote:
No I don't see any dbg log messages there. If you tell me exactly what you're looking for, I can add some.
-d
On 07/28/2008 05:58 PM, William Gropp wrote:
Is there a way to test that setting MPICH_NO_LOCAL=1 works? For example, are there DBG hooks so that I can use the debug features to print out the choice of method for vc's?
Bill
On Jul 28, 2008, at 12:46 PM, Darius Buntinas wrote:
Sure, MPICH_NO_LOCAL or MPICH_ODD_EVEN_CLIQUES environment variables.
-d
On 07/28/2008 11:42 AM, William Gropp wrote:
Ergh, is there a way to do this at runtime? Could there be?
Bill
On Jul 28, 2008, at 11:37 AM, Darius Buntinas wrote:
--enable-nemesis-dbg-nolocal -d
On 07/28/2008 11:32 AM, William Gropp wrote:
Sure, though it will be a hack.
Is there some way to force nemesis to use TCP instead of shared memory for all connections?
Bill
On Jul 25, 2008, at 12:02 PM, Rajeev Thakur wrote:
Bill, Would it be possible to enable this local odd-even option for Nemesis in the old nightly tests? The configure option is --enable-nemesis-dbg-localoddeven.
Rajeev
---------------------------------------------------------------- -------- *From:* [email protected] <mailto:[email protected]> <mailto:[email protected]> [mailto:[email protected]] *On Behalf Of *William Gropp *Sent:* Friday, July 25, 2008 11:31 AM *To:* [email protected] <mailto:mpich2- [email protected]> *Subject:* Re: [mpich2-core] Question about tests
Thanks for the info. I'll still stand by the "if its not tested it doesn't work" position, so it would be good to do at least a basic test (by hand if that's easier) before shipping.
Bill
On Jul 25, 2008, at 9:37 AM, Dave Goodell wrote:
The local-oddeven option should cover most of these concerns. However it is worth noting that if there were certain kinds of bugs in the new(ish) MPIDI_GetIPInterfaces function that they might not be detected on only one node. We might need to set up a nightly joint test between two of the non-breadboard machines if we do want to be sure to cover this. OTOH, I'm not too concerned about it, so I wouldn't make it a high priority.
-Dave
On Jul 25, 2008, at 9:24 AM, Pavan Balaji wrote:
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory). Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
1.1a1 is being run only with the old cron jobs on a single machine with no special odd-even options. Before the release, it would be good to test it on multiple machines just for sanity. We once almost released a version that would not run at all on multiple machines because we never tested it. Rajeev
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Friday, July 25, 2008 9:24 AM To: [email protected] Subject: Re: [mpich2-core] Question about tests
Hi Bill,
We are running nemesis with the local-oddeven option which means that all odd ranks form a shared-memory sub-cluster all even ranks do the same. Communication between an odd and an even rank process happens through the network. So, both should be getting tested. Real two-node testing is better, but we were trying to optimize for high-throughput computing given that we have too few machines and too many tests (takes 3-4 days for each round of all tests).
Truly, we should use the old tests and new tests in conjunction so that we can maximize the number of tests that are run.
-- Pavan
On 07/25/2008 09:12 AM, William Gropp wrote:
The old new testing system that I had started included some tests between different machines to ensure that node-to-node worked. The normal tests usually run on a single machine. Are we regularly running any cross-node tests, and if so, where are they? The tests were those that the wiki page calls "Results from the MPICH2 test suite". (My concern is that Nemesis is detecting that it can used shared memory and we're not testing TCP; alternately, we're always testing TCP and never shared memory).
Bill
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
participants (6)
-
Darius Buntinas -
Dave Goodell -
Pavan Balaji -
Rajeev Thakur -
William Gropp -
William Gropp