As I continue to slog through the website migration, I need to know what to do about the various automated testing that is missing from the new website that we previously had. There are several items: * mpich2 test suite: http://www.mcs.anl.gov/research/projects/mpich2/ testing/ * special tests * configure options tests * code coverage analysis * style checker problems * error message extractor problems * non-conforming global symbols * list of FIXMEs Before I go and spend a lot of time making each one of these work, do we still want all of them? My inclination is yes, but if any of them are especially fragile or otherwise useless, it would be nice to know ahead of time. Also, do any of these have planned replacements? -Dave
All of these are useful. Some of them might be best if they were subsumed into the current testing. For example, the special tests are primarily just runs with less common configure options, to make sure that we test things like no weak symbols and memory leak checks. It would be best is those ran more often anyway. The configure options test is, IIRC, a light-weight way to check that the expected options still work; it is also a robust test. The code coverage tests are very valuable, and something that puts us ahead of many other software efforts, including commercial ones (I know one vendor that is interested in what we do because they still don't do careful coverage tests). The style/error message/non-conforming globals check should help reduce bugs and bug reports. The Fixme list should be addressed at some point, perhaps in 1.1. These scripts are relatively robust. Bill On Jan 30, 2008, at 4:44 PM, Dave Goodell wrote:
As I continue to slog through the website migration, I need to know what to do about the various automated testing that is missing from the new website that we previously had. There are several items:
* mpich2 test suite: http://www.mcs.anl.gov/research/projects/ mpich2/testing/ * special tests * configure options tests * code coverage analysis * style checker problems * error message extractor problems * non-conforming global symbols * list of FIXMEs
Before I go and spend a lot of time making each one of these work, do we still want all of them? My inclination is yes, but if any of them are especially fragile or otherwise useless, it would be nice to know ahead of time. Also, do any of these have planned replacements?
-Dave
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
* mpich2 test suite: http://www.mcs.anl.gov/research/projects/mpich2/testing/ * special tests * configure options tests
These are already present in the new script.
* code coverage analysis * style checker problems * error message extractor problems * non-conforming global symbols * list of FIXMEs
What we have for these is OK for now. They can continue to run on schwinn, etc., since these don't have any problems running on shared machines. The plan is to do the most required functionality in the new script for now and get undergrad interns to port the remaining tests over. If something is straight-forward, let's add it and leave everything else for the interns. Thanks. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji
On Jan 30, 2008, at 5:22 PM, Pavan Balaji wrote:
* mpich2 test suite: http://www.mcs.anl.gov/research/projects/ mpich2/testing/ * special tests * configure options tests
These are already present in the new script.
That's not really true. For example, we aren't running the mpich2 test suite against OS X (darwin) or against the portland group compiler right now. Also, the "mpich2 test suite" referenced above also has links to configure logs and other helpful debugging info. This functionality is not currently present in the new nightly test harness.
* code coverage analysis * style checker problems * error message extractor problems * non-conforming global symbols * list of FIXMEs
What we have for these is OK for now. They can continue to run on schwinn, etc., since these don't have any problems running on shared machines.
I'm not sure that you understood me. These tests may still be running, but their output is going nowhere right now. There's no way to see the results of any of these tests, so we're just as well off as if we weren't even running them right now.
The plan is to do the most required functionality in the new script for now and get undergrad interns to port the remaining tests over. If something is straight-forward, let's add it and leave everything else for the interns.
I wasn't proposing rolling them into the new test scripts, just to modify them such that the results are displayed on the new website. -Dave
That's not really true. For example, we aren't running the mpich2 test suite against OS X (darwin) or against the portland group compiler right now. Also, the "mpich2 test suite" referenced above also has links to configure logs and other helpful debugging info. This functionality is not currently present in the new nightly test harness.
True. We should retain the ones which are not already taken care of by the new tests.
I wasn't proposing rolling them into the new test scripts, just to modify them such that the results are displayed on the new website.
These used to be displayed by the older website. Is it a matter of just redirecting the links? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji
The "mpich2 test suite" refers to the new test harness Bill wrote that can test across 2 machines. It's probably not needed since your new test script can do that. The special tests used to be run manually via /home/MPI/nightly/specialtest. I ran them last on Jan 25th. The getcoverage tests should run 3 times a week based on the foll. cron job on triumph. I am not sure why their output is not visible. 49 06 * * 6 /home/MPI/nightly/getcoverage -updateweb -quiet 49 06 * * 4 /home/MPI/nightly/getcoverage -updateweb -quiet -device=ch3:ssm 49 06 * * 2 /home/MPI/nightly/getcoverage -updateweb -quiet -device=ch3:nemesis I see the line "To include the results of that test in the display on this page, edit todo.txt in the mpich2-web CVS module." under Coverage analysis. Not sure what that means. I believe the style checker, error message extractor, and non-conforming global symbols also used to be run manually, but they haven't been run in a while. Rajeev
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Dave Goodell Sent: Wednesday, January 30, 2008 5:46 PM To: [email protected] Cc: Dave Goodell Subject: Re: [mpich2-core] other automated testing
On Jan 30, 2008, at 5:22 PM, Pavan Balaji wrote:
* mpich2 test suite: http://www.mcs.anl.gov/research/projects/ mpich2/testing/ * special tests * configure options tests
These are already present in the new script.
That's not really true. For example, we aren't running the mpich2 test suite against OS X (darwin) or against the portland group compiler right now. Also, the "mpich2 test suite" referenced above also has links to configure logs and other helpful debugging info. This functionality is not currently present in the new nightly test harness.
* code coverage analysis * style checker problems * error message extractor problems * non-conforming global symbols * list of FIXMEs
What we have for these is OK for now. They can continue to run on schwinn, etc., since these don't have any problems running on shared machines.
I'm not sure that you understood me. These tests may still be running, but their output is going nowhere right now. There's no way to see the results of any of these tests, so we're just as well off as if we weren't even running them right now.
The plan is to do the most required functionality in the new script for now and get undergrad interns to port the remaining tests over. If something is straight-forward, let's add it and leave everything else for the interns.
I wasn't proposing rolling them into the new test scripts, just to modify them such that the results are displayed on the new website.
-Dave
The style checker, error message extractor, and global symbol tests were run by cron on a regular basis. As Dave noted, the new test harness that I wrote included a way to capture the configure output (including the config.log files); something like that will be invaluable if remote runs are submitted. The getcoverage scripts need to update the web site; the output is converted into individual html files. The top level entry needs to know where they are written, and that may have changed. Bill On Jan 30, 2008, at 9:17 PM, Rajeev Thakur wrote:
The "mpich2 test suite" refers to the new test harness Bill wrote that can test across 2 machines. It's probably not needed since your new test script can do that.
The special tests used to be run manually via /home/MPI/nightly/ specialtest. I ran them last on Jan 25th.
The getcoverage tests should run 3 times a week based on the foll. cron job on triumph. I am not sure why their output is not visible.
49 06 * * 6 /home/MPI/nightly/getcoverage -updateweb -quiet 49 06 * * 4 /home/MPI/nightly/getcoverage -updateweb -quiet - device=ch3:ssm 49 06 * * 2 /home/MPI/nightly/getcoverage -updateweb -quiet -device=ch3:nemesis
I see the line "To include the results of that test in the display on this page, edit todo.txt in the mpich2-web CVS module." under Coverage analysis. Not sure what that means.
I believe the style checker, error message extractor, and non- conforming global symbols also used to be run manually, but they haven't been run in a while.
Rajeev
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Dave Goodell Sent: Wednesday, January 30, 2008 5:46 PM To: [email protected] Cc: Dave Goodell Subject: Re: [mpich2-core] other automated testing
On Jan 30, 2008, at 5:22 PM, Pavan Balaji wrote:
* mpich2 test suite: http://www.mcs.anl.gov/research/projects/ mpich2/testing/ * special tests * configure options tests
These are already present in the new script.
That's not really true. For example, we aren't running the mpich2 test suite against OS X (darwin) or against the portland group compiler right now. Also, the "mpich2 test suite" referenced above also has links to configure logs and other helpful debugging info. This functionality is not currently present in the new nightly test harness.
* code coverage analysis * style checker problems * error message extractor problems * non-conforming global symbols * list of FIXMEs
What we have for these is OK for now. They can continue to run on schwinn, etc., since these don't have any problems running on shared machines.
I'm not sure that you understood me. These tests may still be running, but their output is going nowhere right now. There's no way to see the results of any of these tests, so we're just as well off as if we weren't even running them right now.
The plan is to do the most required functionality in the new script for now and get undergrad interns to port the remaining tests over. If something is straight-forward, let's add it and leave everything else for the interns.
I wasn't proposing rolling them into the new test scripts, just to modify them such that the results are displayed on the new website.
-Dave
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
The coverage results actually were getting published, they just had improper linking. I've added a wiki node about this: http:// wiki.mcs.anl.gov/mpich2/index.php/Code_Coverage_Reports. I think that I have finally more pages from the old site moved into the wiki than are available on the newer static site, so I'm going to switch the "for developers" link over to point to the new wiki in the near future. I'll see what I can do about the other tests now. Thanks everyone for info on these various processes. -Dave On Jan 30, 2008, at 9:17 PM, Rajeev Thakur wrote:
The "mpich2 test suite" refers to the new test harness Bill wrote that can test across 2 machines. It's probably not needed since your new test script can do that.
The special tests used to be run manually via /home/MPI/nightly/ specialtest. I ran them last on Jan 25th.
The getcoverage tests should run 3 times a week based on the foll. cron job on triumph. I am not sure why their output is not visible.
49 06 * * 6 /home/MPI/nightly/getcoverage -updateweb -quiet 49 06 * * 4 /home/MPI/nightly/getcoverage -updateweb -quiet - device=ch3:ssm 49 06 * * 2 /home/MPI/nightly/getcoverage -updateweb -quiet -device=ch3:nemesis
I see the line "To include the results of that test in the display on this page, edit todo.txt in the mpich2-web CVS module." under Coverage analysis. Not sure what that means.
I believe the style checker, error message extractor, and non- conforming global symbols also used to be run manually, but they haven't been run in a while.
Rajeev
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Dave Goodell Sent: Wednesday, January 30, 2008 5:46 PM To: [email protected] Cc: Dave Goodell Subject: Re: [mpich2-core] other automated testing
On Jan 30, 2008, at 5:22 PM, Pavan Balaji wrote:
* mpich2 test suite: http://www.mcs.anl.gov/research/projects/ mpich2/testing/ * special tests * configure options tests
These are already present in the new script.
That's not really true. For example, we aren't running the mpich2 test suite against OS X (darwin) or against the portland group compiler right now. Also, the "mpich2 test suite" referenced above also has links to configure logs and other helpful debugging info. This functionality is not currently present in the new nightly test harness.
* code coverage analysis * style checker problems * error message extractor problems * non-conforming global symbols * list of FIXMEs
What we have for these is OK for now. They can continue to run on schwinn, etc., since these don't have any problems running on shared machines.
I'm not sure that you understood me. These tests may still be running, but their output is going nowhere right now. There's no way to see the results of any of these tests, so we're just as well off as if we weren't even running them right now.
The plan is to do the most required functionality in the new script for now and get undergrad interns to port the remaining tests over. If something is straight-forward, let's add it and leave everything else for the interns.
I wasn't proposing rolling them into the new test scripts, just to modify them such that the results are displayed on the new website.
-Dave
participants (4)
-
Dave Goodell -
Pavan Balaji -
Rajeev Thakur -
William Gropp