March 2009 - Mpich2-core - mailman3.cels.anl.gov

Re: [mpich2-core] mpd vs. mpd
by Anthony Chan 30 Mar '09

30 Mar '09

In terms of documentation, one option is that we can de-emphasize the use of mpd.py(or mpd, pmpd, mpddaemon or mpdd) altogether, instead we recommend users to use "mpdboot -n 1" instead of "mpd.py &" to minimize the jolt to users. (Ralph, is "mpdboot -n 1" works as well as "mpd &" ?) Since the music player can be installed before or after mpich2, we are stuck to resolve the name conflict issue. A.Chan ----- "Rusty Lusk" <lusk(a)mcs.anl.gov> wrote: > Ralph and I think we are OK with mpd.py, but doesn't this invalidate > > tons of documentation? Perhaps better would be "mpddaemon" or "mpdd", > > to remain parallel to the other mpd-related executables like mpdcheck, > > mpdtrace, etc. Even then, changing things would be quite a jolt for > > users. Could there be an option to rename mpdd to mpd if you aren't > > using the music player? Perhaps I don't completely understand the > problem. > > On Sunday,Mar 29, 2009, at 10:25 PM, Pavan Balaji wrote: > > > > > I'm actually waiting for Rusty's response on whether we can just > > throw the name 'mpd' out and use 'mpd.py' for the Fedora package. > > That'll be easier. > > > > -- Pavan > > > > Rob Ross wrote: > >> pmpd, p is for python. > >> On Mar 29, 2009, at 10:17 PM, Pavan Balaji wrote: > >>> > >>> Ah, never mind. I had some conversation with Deji and he had > >>> pointed out that this is a problem with rpm -ql, that it doesn't > > >>> show the alternative name. If you do an "mpicc -show", it should > > >>> show the correct version (mpicc should be a link to the > >>> appropriate package's mpicc). > >>> > >>> For mpd, we can't use the same technique, since the mpd music > >>> player also needs to be "alternatives capable" for that. So, we'll > > >>> need to coordinate with the author of that package for it. It's > >>> just easier to either change the name or use a different process > > >>> manager. > >>> > >>> -- Pavan > >>> > >>> Anthony Chan wrote: > >>>> No. > >>>> /home/chan> rpm -q openmpi > >>>> package openmpi is not installed > >>>> ----- "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > >>>>> Do you already have Open MPI installed on your system? > >>>>> > >>>>> -- Pavan > >>>>> > >>>>> Anthony Chan wrote: > >>>>>> I just did a yum search for mpich2 > >>>>>> > >>>>>> [root@localhost proc]# yum search mpich2 > >>>>>> Loaded plugins: refresh-packagekit > >>>>>> =============================== Matched: mpich2 > >>>>> ================================ > >>>>>> mpich2.i386 : An implementation of MPI > >>>>>> mpich2-devel.i386 : Development files for mpich2 > >>>>>> mpich2-libs.i386 : Libraries and configuration files for > mpich2 > >>>>>> > >>>>>> So I installed all 3 rpms through yum and notice: > >>>>>> > >>>>>> /home/chan> rpm -ql mpich2-devel > >>>>>> /usr/bin/mp32-mpicc > >>>>>> /usr/bin/mp32-mpicxx > >>>>>> /usr/bin/mp32-mpif77 > >>>>>> /usr/bin/mp32-mpif90 > >>>>>> > >>>>>> So all MPI compiler wrappers are prefixed with mp32-. Just to > be > >>>>>> consistent, mpd could be prefixed with mp32-. (But I don't > like > >>>>>> any of MPI compiler wrapper prefixed with mp32-) All mpich2's > >>>>> headers > >>>>>> are installed in /usr/include/mpich2/. That explains how > mpich2 > >>>>> isn't > >>>>>> conflicting with openmpi. Also, jumpshot does not work in the > > >>>>>> rpm, > >>>>>> because it uses GNU's broken JVM. Could you ask Deji to make > > >>>>>> sure it uses openjdk or sun's jdk.... ? > >>>>>> > >>>>>> PS. Not sure if this is going to be happened to Deji's > version... > >>>>>> > >>>>>> A.Chan > >>>>>> > >>>>>> ----- "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > >>>>>> > >>>>>>>> Did Deji say that we are the only binary package for MPI on > >>>>> fedora > >>>>>>> ? > >>>>>>> > >>>>>>> Yup. I can try to find his original email, but it was > something > >>>>> along > >>>>>>> the lines of packages have ranks, and MPICH2 is rank 1 > currently > >>>>> since > >>>>>>> there's no other MPI package in there. > >>>>>>> > >>>>>>> -- Pavan > >>>>>>> > >>>>>>> -- > >>>>>>> Pavan Balaji > >>>>>>> http://www.mcs.anl.gov/~balaji > >>>>> -- > >>>>> Pavan Balaji > >>>>> http://www.mcs.anl.gov/~balaji > >>> > >>> -- > >>> Pavan Balaji > >>> http://www.mcs.anl.gov/~balaji > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji

2 1

Re: [mpich2-core] mpd vs. mpd
by Anthony Chan 30 Mar '09

30 Mar '09

I just did a yum search for mpich2 [root@localhost proc]# yum search mpich2 Loaded plugins: refresh-packagekit =============================== Matched: mpich2 ================================ mpich2.i386 : An implementation of MPI mpich2-devel.i386 : Development files for mpich2 mpich2-libs.i386 : Libraries and configuration files for mpich2 So I installed all 3 rpms through yum and notice: /home/chan> rpm -ql mpich2-devel /usr/bin/mp32-mpicc /usr/bin/mp32-mpicxx /usr/bin/mp32-mpif77 /usr/bin/mp32-mpif90 So all MPI compiler wrappers are prefixed with mp32-. Just to be consistent, mpd could be prefixed with mp32-. (But I don't like any of MPI compiler wrapper prefixed with mp32-) All mpich2's headers are installed in /usr/include/mpich2/. That explains how mpich2 isn't conflicting with openmpi. Also, jumpshot does not work in the rpm, because it uses GNU's broken JVM. Could you ask Deji to make sure it uses openjdk or sun's jdk.... ? PS. Not sure if this is going to be happened to Deji's version... A.Chan ----- "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > > Did Deji say that we are the only binary package for MPI on fedora > ? > > Yup. I can try to find his original email, but it was something along > > the lines of packages have ranks, and MPICH2 is rank 1 currently since > > there's no other MPI package in there. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

4 7

Re: [mpich2-core] mpd vs. mpd
by Anthony Chan 29 Mar '09

29 Mar '09

Did Deji say that we are the only binary package for MPI on fedora ? It sounds like getting rid of mpd and use mpd.py directly for the fedora binary package is reasonable, should confirm with Rusty/Ralph to see what they think just in case if we miss any important reason for "mpd". A.Chan ----- "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > Deji, who packages our Fedora builds, said that right now we are the > only binary package for MPICH2. Maybe he was mistaken, or maybe he was > > referring to something else. > > If the mpd* commands don't use mpd directly, why not get rid of it and > > only have mpd.py? > > -- Pavan > > Anthony Chan wrote: > > I thought OpenMPI is already in most linux distributions. I can > > do a "yum install openmpi" on my fc9. > > > > If there isn't any better solution, I guess we can rename mpd as > > mpich2-mpd. AFAIK, none of the mpd* command uses "mpd" command > > (it uses mpd.py, needs to confirm with Ralph). But then we need > > to update all our documentations about mpich2-mpd.... > > > > ----- "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > > > >> They have some model called "alternatives" for equivalent packages > > >> (e.g., all MPI packages); I don't exactly know how it works. > Besides, > >> > >> since we are in first, Open MPI can deal with those issues :-). In > the > >> > >> same way, since the music player went in first, we need to figure > out > >> > >> how to add mpd in there. > >> > >> -- Pavan > >> > >> Anthony Chan wrote: > >>> Is fedora putting mpich2's mpd and the music player daemon > >>> in /usr/bin ? If so, how do they resolve MPI wrappers issue > >>> that OpenMPI and MPICH2 both have identical MPI compiler > >>> wrapper names (or worse they both have > mpi.h/mpif.h/mpicxx.h....)? > >>> I would think that whatever the solution they used for OpenMPI > >>> vs MPICH2 should be applicable in mpd vs mpd. > >>> > >>> A.Chan > >>> > >>> ----- "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > >>> > >>>> Our Fedora binary package maintainer sent me an email about the > >>>> conflicting name of "mpd". (we already have a ticket for this: > >>>> https://trac.mcs.anl.gov/projects/mpich2/ticket/165) > >>>> > >>>> The basic problem is that there's some random music player that > is > >>>> also > >>>> called "mpd" (which is already in the Fedora distributions) and > >> this > >>>> is > >>>> causing problems for the Fedora binary package for MPICH2 to > >> break. > >>>> What > >>>> options do we have here? Can we easily change the name of the > "mpd" > >>>> daemons to something else, e.g., "mpd-is-not-mpd" (mpdboot can > >> still > >>>> have the same name)? Should we ask them to configure with > remshell > >>>> instead (it's 1.0.8, so no Hydra in there yet)? > >>>> > >>>> This is a show-stopper for MPICH2's acceptance into FC11. > >>>> > >>>> -- Pavan > >>>> > >>>> -- > >>>> Pavan Balaji > >>>> http://www.mcs.anl.gov/~balaji > >> -- > >> Pavan Balaji > >> http://www.mcs.anl.gov/~balaji > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

2 1

Re: [mpich2-core] mpd vs. mpd
by Anthony Chan 29 Mar '09

29 Mar '09

I thought OpenMPI is already in most linux distributions. I can do a "yum install openmpi" on my fc9. If there isn't any better solution, I guess we can rename mpd as mpich2-mpd. AFAIK, none of the mpd* command uses "mpd" command (it uses mpd.py, needs to confirm with Ralph). But then we need to update all our documentations about mpich2-mpd.... ----- "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > They have some model called "alternatives" for equivalent packages > (e.g., all MPI packages); I don't exactly know how it works. Besides, > > since we are in first, Open MPI can deal with those issues :-). In the > > same way, since the music player went in first, we need to figure out > > how to add mpd in there. > > -- Pavan > > Anthony Chan wrote: > > Is fedora putting mpich2's mpd and the music player daemon > > in /usr/bin ? If so, how do they resolve MPI wrappers issue > > that OpenMPI and MPICH2 both have identical MPI compiler > > wrapper names (or worse they both have mpi.h/mpif.h/mpicxx.h....)? > > I would think that whatever the solution they used for OpenMPI > > vs MPICH2 should be applicable in mpd vs mpd. > > > > A.Chan > > > > ----- "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > > > >> Our Fedora binary package maintainer sent me an email about the > >> conflicting name of "mpd". (we already have a ticket for this: > >> https://trac.mcs.anl.gov/projects/mpich2/ticket/165) > >> > >> The basic problem is that there's some random music player that is > >> also > >> called "mpd" (which is already in the Fedora distributions) and > this > >> is > >> causing problems for the Fedora binary package for MPICH2 to > break. > >> What > >> options do we have here? Can we easily change the name of the "mpd" > > >> daemons to something else, e.g., "mpd-is-not-mpd" (mpdboot can > still > >> have the same name)? Should we ask them to configure with remshell > > >> instead (it's 1.0.8, so no Hydra in there yet)? > >> > >> This is a show-stopper for MPICH2's acceptance into FC11. > >> > >> -- Pavan > >> > >> -- > >> Pavan Balaji > >> http://www.mcs.anl.gov/~balaji > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

2 1

mpd vs. mpd
by Pavan Balaji 29 Mar '09

29 Mar '09

Our Fedora binary package maintainer sent me an email about the conflicting name of "mpd". (we already have a ticket for this: https://trac.mcs.anl.gov/projects/mpich2/ticket/165) The basic problem is that there's some random music player that is also called "mpd" (which is already in the Fedora distributions) and this is causing problems for the Fedora binary package for MPICH2 to break. What options do we have here? Can we easily change the name of the "mpd" daemons to something else, e.g., "mpd-is-not-mpd" (mpdboot can still have the same name)? Should we ask them to configure with remshell instead (it's 1.0.8, so no Hydra in there yet)? This is a show-stopper for MPICH2's acceptance into FC11. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

2 2

mpich2 on niagara
by Darius Buntinas 26 Mar '09

26 Mar '09

Just an FYI, MPICH2 trunk now works on niagara. I'm seeing 5us latencies. I was concerned until I saw that open mpi is getting 8us. This still seems way too high, and is something we should probably look into. -d

1 0

Re: [mpich2-core] simplemake vs. automake
by Anthony Chan 25 Mar '09

25 Mar '09

Here is my $0.02 1) flexibility: we can modify simplemake to add things like Pretty Make. We don't have to depend on automake developers to fix the problems for us. 2) automake seems to rely on libtool which is the main problem for me. Correct me if I am wrong. It seems that using automake can be equated to our willingness to using libtool in create static and shared libraries. I could be biased (i.e. less experienced in libtool), I feel like supporting libtool is more difficult than supporting autoconf.... If we think the original Makefile output from simplemake is horrible (before Pretty Make), adding libtool in the equations is 10 times worse.... Also, the build will likely take much longer. 3) It sounds like you are suggesting that automake has native support for the timestamp issue on NFS. I think the issue has to do with Makefile rules and timestamp resolution of the file than the simplemake vs automake. I feel like I am against automake, but mainly I don't know automake well enough to say that it will benefit us. A.Chan ----- "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > Hi all, > > I know that we have had this discussion at some point, so sorry to > bring > this up again, but can someone point out why exactly we are staying > away > from automake and still using simplemake? This email is mainly so that > > we understand what the problem is, as well as see if there are > solutions > that are available today which weren't available a decade ago when we > > went with simplemake. > > As a downside of simplemake, we have no help from outside as all > system > administrators and more recently binary package maintainers I have > been > in contact with have no idea how to us it. This also keeps biting us > when someone has a problem with timestamps on their NFS and configure > > decides that some things are outdated and tries to run simplemake > again. > (e.g., ticket #461). > > I know that moving to automake is effort on our part (which is an > important consideration), but let's ignore that part for the time > being > and just concentrate on the other technical difficulties. I'd prefer > to > let others maintain these packaging tools as much as possible, unless > > there is a good reason for us to maintain it ourselves. > > Thanks, > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

1 0

simplemake vs. automake
by Pavan Balaji 24 Mar '09

24 Mar '09

Hi all, I know that we have had this discussion at some point, so sorry to bring this up again, but can someone point out why exactly we are staying away from automake and still using simplemake? This email is mainly so that we understand what the problem is, as well as see if there are solutions that are available today which weren't available a decade ago when we went with simplemake. As a downside of simplemake, we have no help from outside as all system administrators and more recently binary package maintainers I have been in contact with have no idea how to us it. This also keeps biting us when someone has a problem with timestamps on their NFS and configure decides that some things are outdated and tries to run simplemake again. (e.g., ticket #461). I know that moving to automake is effort on our part (which is an important consideration), but let's ignore that part for the time being and just concentrate on the other technical difficulties. I'd prefer to let others maintain these packaging tools as much as possible, unless there is a good reason for us to maintain it ourselves. Thanks, -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

icrunch, http
by Anthony Chan 19 Mar '09

19 Mar '09

I have just put a copy of 1.0.8 and 1.1b1 in our ftp servers. On icrunch, there isn't wget but only ftp, so it isn't easy to get a copy of our latest 1.1b1 or even 1.0.8. I think there are machines that doesn't come with wget/lynx but plain old ftp command besides Mac. We should consider putting a copy of mpich2 releases on our ftp server. A.Chan

2 2

Re: [mpich2-core] Moving confdb to a separate repository
by Anthony Chan 19 Mar '09

19 Mar '09

----- "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > svn:externals is like a symbolic link > (http://svnbook.red-bean.com/en/1.0/ch07s03.html) sounds kind of like cvs module. Will need to read it more thoroughly. > Is it possible to move out all the common ones to svn:externals. > and have MPE specific ones in the MPE directory? Are you still talking about m4 files ? The duplicated PAC macros are not in the same file structure as MPICH2's, so it isn't a matter of removing duplicate files. One needs to compare the macros one by one... The process could be tedious, so I still haven't gotten a chance to clean up the MPE macros w.r.t. MPICH2's since the idea of consolidating all m4 macros into confdb was first discussed. Another issue is that if I move all common PAC macros to MPICH2, that means standalone MPE needs to checkout MPICH2's condfb in a directory within MPE for MPE's maint/updatefiles to work. Hmm... that could mean the checkout version of MPICH2 has 2 copies of confdb. A.Chan > > -- Pavan > > Anthony Chan wrote: > > the slog2sdk/trace_xxx/confdb/*.m4 are new libtool m4 macros. > > These macros are NOT duplicates of mpich2/confdb/*.m4. > > I plan to create a mpe2/confdb, again this won't be the same > > as mpich2/confdb. > > > > Not sure what svn:externals is. If you are going to move > > mpich2/confdb to svn:externals. Should we also move mpe2 > > to svn:externals as well... > > > > > > ----- "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > > > >> Hi all, > >> > >> Currently confdb is present in four different locations in the > mpich2 > >> stack: > >> > >> mpi/mpich2/trunk/confdb > >> mpi/mpich2/trunk/src/mpi/romio/confdb > >> mpi/mpich2/trunk/src/slog2sdk/trace_sample/confdb > >> mpi/mpich2/trunk/src/slog2sdk/trace_rlog/confdb > >> > >> Is there a reason we need to maintain four copies this way? Note > that > >> > >> when we had a CVS, this was all just one repository that used to > get > >> dynamically checked out in different places. We initially > considered > >> doing that with svn too using svn:externals, but as Darius > discovered, > >> > >> commits don't recurse into svn:external directories. However, since > > >> confdb is anyway considered a separate piece of code, does it makes > > >> sense to do it now? > >> > >> RobL already confirmed that he was OK with this change. Anthony, > can > >> you > >> confirm this is OK with you too? Or if others have any objections, > > >> please let me know. > >> > >> Thanks, > >> > >> -- Pavan > >> > >> -- > >> Pavan Balaji > >> http://www.mcs.anl.gov/~balaji > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

2 1