March 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Problems with running make
by Ron Palmer 02 Mar '14

02 Mar '14

Gus, following my reply to Reut's email, I thought I will clarify some of the details of my system. I do not have control over the actual software to be run on the cluster. I have asked whether it requires fortran or not but have not yet received any answers. I do know that they are running the very same version of this application on a mpi 1.4.1 cluster. I have those two PATH statements (PATH and LD_LIBRARY_PATH) in each of the three computers /home/pgcinversion. Each of the computers have their own users, separate home directories, I just created the same username on all three of them, pgcinversion. sargeant is the master, most hdd space, and has exported /inv, which is then mounted via nfs in fstab by the others. constable and gainborough are slaves all three can connect to any of the other two without ssh password I am not sure whether I have compiled with shared libraries, though in my reply to reuti I had screenshots of the output of ldd cpi on the three computers, will that answer your question? in regards to compiling cpi, I used your suggestion of mpicc -o cpi cpi.c mpiexec -machinefile all_machines -np 24 ./cpi should I modify and re-compile? I am really unsure about how to set it up. I saved the version of cpi that came with the download as cpi_orig, and then ran the one I compiled (cpi) and the one that came with the download, cpi_orig. A previous email of mine has those screenshots (happy to copy and paste again if you prune your email history). NOTE: You *must install on ALL computers* otherwise the runtime will be missing in some of them. Note that rpms install on *local* directories (/usr and friends), not on you NFS share (as opposed to what you did with MPICH). Was this the issue? Gus, as Reuti suspected, there was (and may still be) a lack of libraries installed, see my reply to Reuti 30min ago. Installing compilers and getting all parts right is my weakest link, and where I had guess a bit at the start (is that called learning?????). Once I hear back from the list on the outputs of ldd cpi, I will install all listed by yum list | grep gfortran. I have shut down iptables on sargeant, but keep them running on the other two . I will log on to the cluster, shut them down, try cpi again and report back. Does anyone have a good itables command to include to allow me to run iptables and mpi cluster applications? Please fire off as may questions as you like, am learning a lot and we are getting closer! Greatly appreciated. Ron On 3/03/2014 02:43, Gustavo Correa wrote: > Hi Ron > > On Mar 2, 2014, at 7:20 AM, Ron Palmer wrote: > >> Gus, Reuti, Rajeev and others, >> I do have mpif77 and mpif90 in mpi/bin, so I conclude that it has compiled ok. >> > It looks like it compiled the F77 and F90 MPICH interfaces with gfortran. > >> I have checked reuti's suggestions, indicating that the environment is ok. I had added 'PATH=/my/mpi/bin:$PATH' to each of the .bashrc files (for the user I am running on all three) >> > Absolutely, you must setup the environment, PATH in particular, if MPICH is not > installed on a standard directory searched by Linux by default. > > Just for our understanding: > Separate home directories on each computer, or a single home on NFS share? > >> I am using the same username on all three computers - is that ok? >> > Rajeev may correct me if I am wrong, > but I think you *must* run MPI as the same user (name and UID) everywhere. > > >> I have installed MPI on to a NFS drive, all accessing the same; 'which' confirms. Are there any libraries that needs to be installed elsewhere? do need to run make install as root? >> > Great. That is the simplest thing to do in a small cluster/set of computers. > >> I have added the 'LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/my/mpi/lib', though made no difference. (should those "{" and "}" be included? I entered it exactly like I have typed) >> > The curly braces are just to protect the string from shell overinterpretation. > You can remove them if you want, probably won't make any difference. > LD_LIBRARY_PATH is important if you built MPICH with shared libraries. > Check the MPICH lib directory for .so files, or do ldd cpi and see if there is any MPICH > shared libraries there. > >> I have mpicc the cpi program as suggested by Gus and tried it, as well as the originally provided cpi (part of the downloaded mpi) which I called cpi_orig, see the screen shot below. It looks like there is an issue with libgfortran.so.3... Perhaps I have not got the various compilers installed correctly? I would appreciate if someone could give me a suggestion on how to properly remove and re-install these (and select the correct ones). >> > A) I am rather surprised that libgfortran is playing a role in cpi, which presumably is a C program. > Did you compile it with mpicc? > Did you compile the C cpi.c program, or did you compile a different program, > perhaps a Fortran version of it? > > B) Having said that, I concur with what Reuti already told you. > Your gfortran installation may not be complete, it may lack runtime libraries, maybe > some include files. > Some times the Linux distribution packs them in separate rpms. > Try "yum list | grep gfortran", and install anything that you may not have. > > NOTE: You *must install on ALL computers* otherwise the runtime will be missing > in some of them. > Note that rpms install on *local* directories (/usr and friends), not on you NFS share > (as opposed to what you did with MPICH). > Was this the issue? > > So far you didn't say if the actual program you need to run is Fortran or C. > If it is Fortran you definitely need a Fortran compiler (gfortran) fully installed on all nodes. > What is it? > > >> By the way, in the runs of cpi below, I reduced the number of cores to be used on the first pc listed in the machinefile, from 8 to 4, in the case there was an issue with the 9th one. As can be seen, it is the first node (or core) on a remote computer where the problems start to be listed which is node 5 in this case... >> > I wild-guessed there might be a problem with rank 9, but that was only in *your program*. > In any case, the printout you sent before with your program output may have just > issued a red herring, not so important. > > Let's concentrate on cpi for now. > If you get cpi running, it is a big leap forward, because after that you can > be sure MPICH is working with all bells and whistles. > Divide and conquer, so to speak. > Then you can address the issues with your program, which is a separate problem. > >> If I run cpi on only one computer, it works. And yes, hostname works even if all computers are included in the machinefile. >> > I would guess something is blocking the launching of mpi tasks on the other computers. > Besides the head computer (seargent ?), where you seem to have turned off the firewall' > already, do you have firewalls in the *other* computers? > Any other access restrictions (on /hosts/deny /hosts/allow or other)? > Can you ssh passwordless *back and forth* across *any pair* of computers, > using the same user name and UID that you use to launch cpi and other MPI programs? > >> Clear as mud, hey? >> > Cristal clear, as any issue with parallel programs. :) > > I hope it helps, > Gus Correa > >> Thanks for your help. >> >> Ron >> >> >> <ehddjihc.png> >> >> >> On 2/03/2014 05:20, Gustavo Correa wrote: >>> Hi Ron >>> >>> Please, see my comments inline. >>> >>> On Mar 1, 2014, at 6:29 AM, Reuti wrote: >>> >>> >>>> Hi, >>>> >>>> Am 01.03.2014 um 12:18 schrieb Ron Palmer: >>>> >>>> >>>>> Gus and others, >>>>> many thanks for your comments and suggestions. I am unsure whether I need fortran included as I am not in control of the software to be run on this linux cluster. I have asked but not yet got a reply... >>>>> >>>>> However, I did a yum search gfortran and installed the gcc-gfortran.x86_64 (I hope that was the one you referred to). I followed up with the ./configure ... as per your suggestion. That did not work. I then tried with only ./configure --prefix=/my/bin/folder. That worked so I also did make and make install. >>>>> >>>>> >>> Maybe there was a misspell in the configure command line with >>> I am not 100% sure about the required gfortran yum RPMs. >>> It may also need the corresponding "devel" package, if you want to check, >>> but I am not really sure. Maybe you already installed everything you need for >>> a functional gfortran. >>> >>> As far as I can tell from your c2.txt file, configure seems to have recognized gfortran. >>> I can't tell for sure if it actually built the MPICH F77 and F90, although chances are that it did. >>> However, you may check if you have mpif77 and mpif90 in the MPICH bin directory, >>> which would be a good sign. >>> >>> I would not leave the F77 and F90 interfaces out of the game, >>> because sooner or later somebody will come with a Fortran MPI program >>> to run on your machine. >>> >>> >>>>> I have a machinefile with three linux computers, all sharing the same mpi binaries. Running hostname on all 20 cores works, so I guess I should conclude that mpi works, right? >>>>> >>>> Are they using the same binaries /my/bin/folder for a non-interactive startup too, i.e.: >>>> >>>> $ ssh sargeant which hydra_pmi_proxy >>>> >>>> returns the same as on the master (maybe ~/.bashrc needs to set the $PATH)? >>>> >>>> -- Reuti >>>> >>>> >>> Make sure you check Reuti's suggestions. >>> >>> Did you install MPICH in an NFS shared (across all machines) directory, >>> or did you install on each machine in a local directory? >>> >>> As Reuti pointed out, the environment setup is important. >>> If MPICH is not installed in a directory that Linux searches by default, such as /usr or /usr/local, >>> you need to add the MPICH bin directory to the PATH, and probably the lib directory to >>> LD_LIBRARY_PATH (if built with shared libraries). >>> The simplest way is in the .bashrc/.tcshrc file in the user home directory (which I presume >>> is shared via NFS across the computers, otherwise you need to edit these files on each >>> computer) >>> Say: >>> >>> - bash: >>> >>> export PATH=${PATH}:/my/mpich/bin >>> export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/my/mpich/lib >>> >>> - tcsh: >>> >>> setenv PATH ${PATH}:/my/mpich/bin >>> setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/my/mpich/lib >>> >>> Also, you can go a long way troubleshooting these problems if you read the MPICH >>> documentation (Installer's Guide, User's Guide, README file) >>> >>> >>> http://www.mpich.org/documentation/guides/ >>> >>> >>> If you really could run mpiexec hostname across all computers, I would guess you >>> don't have any security or network related issues (i.e. passwordless ssh works >>> across the nodes, there is no firewall blocking, the nodes names are properly >>> resolved most likely via /etc/hosts, and so on). >>> >>> To make sure MPICH works with a simple MPI program, >>> compile the cpi.c program in the examples directory (comes with the source code tarball), >>> and run it across all nodes. >>> >>> Something like this: >>> >>> mpicc -o cpi cpi.c >>> mpiexec -machinefile all_machines -np 24 ./cpi >>> >>> If it runs, then MPICH is alright, and any issues you are having with the other program, >>> are not an MPICH issue, but a bug or misconfiguration in the other program. >>> >>> >>> >>>>> I then tried the software I am supposed to run on the cluster. It works on all nodes on the master computer, but not if other computers are involved. I have captured the output from such failed attempt to run that software on all available nodes on all available computers in the cluster, see below. I have attached c2.txt and m2.txt (and the output of the failed run) in the attached gzipped file (the failed run in run2.txt and in the image below may not be from the same failed run...). >>>>> >>>>> >>>>> So, is it an issue for the software developers or is it an issue with mpich? >>>>> >>>>> >>> The screenshot you sent MPI ranks 9, 11,12, 14 don't printout their ranks. >>> This may or may not indicate a programming error, however combined with the fact that >>> the program fails in MPI_Barrier, which is a collective call, suggests it may be a bug in the program. >>> One possibility is that the MPI program is wrong, and somehow ranks 9,11,12,14 >>> did not call MPI_Barrier (collective calls, as the name says, are collective, >>> must be called by all ranks in the "communicator"). >>> >>> Another line of search would be that in both the screenshot and the run2.txt file >>> complain of "Communication error with rank 8" in the MPI_Barrier. >>> In each case rank 8 is in a different computer (gainsborough and constable). >>> It may be that rank 8 never called MPI_Barrier somehow. >>> For instance, MPI collective calls inside "if" conditionals can cause traps like this, >>> as the "if" condition may turn out to be true on some ranks and false on others. >>> >>> Judging from the small amount of output, the program seems to have failed >>> very early, which may help search for a possible bug (early in the code). >>> >>> However, these are just wild guesses. >>> >>> Is this an in-house program, public domain, or commercial software? >>> Anyway, this may be an issue to bring to that code developers. >>> >>> I hope this helps, >>> Gus Correa >>> >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- *Ron Palmer*MSc MBA. Principal Geophysicist ron.palmer(a)pgcgroup.com.au <mailto:[email protected]> 0413 579 099 07 3103 4963

1 0

Re: [mpich-discuss] Problems with running make
by Ron Palmer 02 Mar '14

02 Mar '14

Reuti, thanks for you suggestions... I did the ldd cpi (in the examples directory) on three computers, and I have attached three screenshot. I am using sargeant as the master, where I start these jobs from. Looking at these, I installed gcc-gfortran.x86_64 on the other two computers as well, and they now have identical outputs as the first one (sargeant). Thanks for pointng this out and giving me help in how to check this. I tried: mpiexec -machinefile cluster.txt -np 6 ./cpi were the machine file has only two members, each with 4 cores. I received the same error message as before but that could be for many reasons, and I will now respond to Gus's email with a few clarifications. I appreciate your efforts! R On 3/03/2014 00:28, Reuti wrote: > Am 02.03.2014 um 14:01 schrieb Ron Palmer: > >> I am running centos on two of these, sargeant and constable, and RHEL on the third, gainsborough. >> >> By runtime libraries, do you mean for the three compilers included, ie gcc, fortran77 and fortran90? > Yes. But there should be a package with the runtime libraries, i.e. so that it's not necessary to install the complete compiler suite (I assume in your situation that they are installed, but with different versions). Maybe you can check for older/newer libs in the repository of the distributions. > > Which libraries are necessary can be listed by issuing: > > $ ldd cpi > > on each of the machines and whether they are found. > > -- Reuti > > NB: With commercial compilers it's for this reason often allowed in the license to give away the runtime libraries (and only these) with your compiled application, as otherwise your customers couldn't run your applications. > > >> Please use one syllable words, I am the edge of my understanding :-(. >> >> Thanks >> Ron >> >> >> On 2/03/2014 22:55, Reuti wrote: >>> Hi, >>> >>> Am 02.03.2014 um 13:20 schrieb Ron Palmer: >>> >>>> Gus, Reuti, Rajeev and others, >>>> I do have mpif77 and mpif90 in mpi/bin, so I conclude that it has compiled ok. >>>> >>>> I have checked reuti's suggestions, indicating that the environment is ok. I had added 'PATH=/my/mpi/bin:$PATH' to each of the .bashrc files (for the user I am running on all three) >>>> >>>> I am using the same username on all three computers - is that ok? >>>> >>>> I have installed MPI on to a NFS drive, all accessing the same; 'which' confirms. Are there any libraries that needs to be installed elsewhere? do need to run make install as root? >>> No. >>> >>> >>>> I have added the 'LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/my/mpi/lib', though made no difference. (should those "{" and "}" be included? I entered it exactly like I have typed) >>>> >>>> I have mpicc the cpi program as suggested by Gus and tried it, as well as the originally provided cpi (part of the downloaded mpi) which I called cpi_orig, see the screen shot below. It looks like there is an issue with libgfortran.so.3... >>> Yep. >>> >>> Unless you build static applications, the runtime libraries (of the compiler) need to be available on all machines. >>> >>> There should be package with the runtime libraries in your Linux distribution. >>> >>> -- Reuti >>> >>> >>>> Perhaps I have not got the various compilers installed correctly? I would appreciate if someone could give me a suggestion on how to properly remove and re-install these (and select the correct ones). >>>> >>>> By the way, in the runs of cpi below, I reduced the number of cores to be used on the first pc listed in the machinefile, from 8 to 4, in the case there was an issue with the 9th one. As can be seen, it is the first node (or core) on a remote computer where the problems start to be listed which is node 5 in this case... >>>> >>>> If I run cpi on only one computer, it works. And yes, hostname works even if all computers are included in the machinefile. >>>> >>>> Clear as mud, hey? >>>> Thanks for your help. >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- *Ron Palmer*MSc MBA. Principal Geophysicist ron.palmer(a)pgcgroup.com.au <mailto:[email protected]> 0413 579 099 07 3103 4963

1 0

Re: [mpich-discuss] Problems with running make
by Reuti 02 Mar '14

02 Mar '14

Hi, Am 02.03.2014 um 22:46 schrieb Gustavo Correa: > Hi Pavan > > Many thanks for this clarification. > I stand corrected. > > Actually I guessed you MPICH developers might have already set > something up to allow this kind of "collaborative" or "dialogue" MPI style > with different users names/IDs running MPI processes across different computers, > for specific uses. > > Not trying to hijack Ron's thread (he has more important hurdles to get behind him), > I wonder how mpiexec (Hydra), which is launched on one computer under > one user name, will trigger additional MPI processes on the other computers > under different user names (and how ssh may have to be setup to allow this to happen). It's necessary to put your public key into the authorized_keys file under the other accounts on the slave machines. I have different accounts at several remote machines and to ease the access, I also put the target account in my ~/.ssh/config like: $ cat ~/.ssh/config Host slave1 User foo Host slave2 User baz Then it will use the specified account automatically. AFAICS it also supports running MPICH applications with different account on a cluster this way without any support by MPICH. == You can also use hostbased authentication, for which I wrote a Howto some time ago: https://arc.liv.ac.uk/SGE/howto/hostbased-ssh.html In case of a different account it must be granted by a setting in ~/.rhosts resp. ~/.shosts in addition. -- Reuti > Cheers, > Gus > > On Mar 2, 2014, at 12:39 PM, Balaji, Pavan wrote: > >> Gus, >> >> On Mar 2, 2014, at 10:43 AM, Gustavo Correa <gus(a)ldeo.columbia.edu> wrote: >>> Rajeev may correct me if I am wrong, >>> but I think you *must* run MPI as the same user (name and UID) everywhere. >> >> Not necessarily. >> >> http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Using… >> >> But using the same username is definitely easier and safer. >> >> — Pavan >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Problems with running make
by Balaji, Pavan 02 Mar '14

02 Mar '14

On Mar 2, 2014, at 3:46 PM, Gustavo Correa <gus(a)ldeo.columbia.edu> wrote: > Not trying to hijack Ron's thread (he has more important hurdles to get behind him), > I wonder how mpiexec (Hydra), which is launched on one computer under > one user name, will trigger additional MPI processes on the other computers > under different user names (and how ssh may have to be setup to allow this to happen). mpiexec launches additional helper processes (proxy) on each “node”. The helper processes can be of any user id. “Node” is just a domain that is controlled by one proxy. The user can treat a single physical nodes as multiple “nodes” in Hydra’s terminology by using different entries in the host file to specify each of them. — Pavan

1 0

Re: [mpich-discuss] Problems with running make
by Gustavo Correa 02 Mar '14

02 Mar '14

Hi Pavan Many thanks for this clarification. I stand corrected. Actually I guessed you MPICH developers might have already set something up to allow this kind of "collaborative" or "dialogue" MPI style with different users names/IDs running MPI processes across different computers, for specific uses. Not trying to hijack Ron's thread (he has more important hurdles to get behind him), I wonder how mpiexec (Hydra), which is launched on one computer under one user name, will trigger additional MPI processes on the other computers under different user names (and how ssh may have to be setup to allow this to happen). Cheers, Gus On Mar 2, 2014, at 12:39 PM, Balaji, Pavan wrote: > Gus, > > On Mar 2, 2014, at 10:43 AM, Gustavo Correa <gus(a)ldeo.columbia.edu> wrote: >> Rajeev may correct me if I am wrong, >> but I think you *must* run MPI as the same user (name and UID) everywhere. > > Not necessarily. > > http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Using… > > But using the same username is definitely easier and safer. > > — Pavan > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problems with running make
by Balaji, Pavan 02 Mar '14

02 Mar '14

Gus, On Mar 2, 2014, at 10:43 AM, Gustavo Correa <gus(a)ldeo.columbia.edu> wrote: > Rajeev may correct me if I am wrong, > but I think you *must* run MPI as the same user (name and UID) everywhere. Not necessarily. http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Using… But using the same username is definitely easier and safer. — Pavan

1 0

Re: [mpich-discuss] Problems with running make
by Gustavo Correa 02 Mar '14

02 Mar '14

Hi Ron On Mar 2, 2014, at 7:20 AM, Ron Palmer wrote: > Gus, Reuti, Rajeev and others, > I do have mpif77 and mpif90 in mpi/bin, so I conclude that it has compiled ok. > It looks like it compiled the F77 and F90 MPICH interfaces with gfortran. > I have checked reuti's suggestions, indicating that the environment is ok. I had added 'PATH=/my/mpi/bin:$PATH' to each of the .bashrc files (for the user I am running on all three) > Absolutely, you must setup the environment, PATH in particular, if MPICH is not installed on a standard directory searched by Linux by default. Just for our understanding: Separate home directories on each computer, or a single home on NFS share? > I am using the same username on all three computers - is that ok? > Rajeev may correct me if I am wrong, but I think you *must* run MPI as the same user (name and UID) everywhere. > I have installed MPI on to a NFS drive, all accessing the same; 'which' confirms. Are there any libraries that needs to be installed elsewhere? do need to run make install as root? > Great. That is the simplest thing to do in a small cluster/set of computers. > I have added the 'LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/my/mpi/lib', though made no difference. (should those "{" and "}" be included? I entered it exactly like I have typed) > The curly braces are just to protect the string from shell overinterpretation. You can remove them if you want, probably won't make any difference. LD_LIBRARY_PATH is important if you built MPICH with shared libraries. Check the MPICH lib directory for .so files, or do ldd cpi and see if there is any MPICH shared libraries there. > I have mpicc the cpi program as suggested by Gus and tried it, as well as the originally provided cpi (part of the downloaded mpi) which I called cpi_orig, see the screen shot below. It looks like there is an issue with libgfortran.so.3... Perhaps I have not got the various compilers installed correctly? I would appreciate if someone could give me a suggestion on how to properly remove and re-install these (and select the correct ones). > A) I am rather surprised that libgfortran is playing a role in cpi, which presumably is a C program. Did you compile it with mpicc? Did you compile the C cpi.c program, or did you compile a different program, perhaps a Fortran version of it? B) Having said that, I concur with what Reuti already told you. Your gfortran installation may not be complete, it may lack runtime libraries, maybe some include files. Some times the Linux distribution packs them in separate rpms. Try "yum list | grep gfortran", and install anything that you may not have. NOTE: You *must install on ALL computers* otherwise the runtime will be missing in some of them. Note that rpms install on *local* directories (/usr and friends), not on you NFS share (as opposed to what you did with MPICH). Was this the issue? So far you didn't say if the actual program you need to run is Fortran or C. If it is Fortran you definitely need a Fortran compiler (gfortran) fully installed on all nodes. What is it? > By the way, in the runs of cpi below, I reduced the number of cores to be used on the first pc listed in the machinefile, from 8 to 4, in the case there was an issue with the 9th one. As can be seen, it is the first node (or core) on a remote computer where the problems start to be listed which is node 5 in this case... > I wild-guessed there might be a problem with rank 9, but that was only in *your program*. In any case, the printout you sent before with your program output may have just issued a red herring, not so important. Let's concentrate on cpi for now. If you get cpi running, it is a big leap forward, because after that you can be sure MPICH is working with all bells and whistles. Divide and conquer, so to speak. Then you can address the issues with your program, which is a separate problem. > If I run cpi on only one computer, it works. And yes, hostname works even if all computers are included in the machinefile. > I would guess something is blocking the launching of mpi tasks on the other computers. Besides the head computer (seargent ?), where you seem to have turned off the firewall' already, do you have firewalls in the *other* computers? Any other access restrictions (on /hosts/deny /hosts/allow or other)? Can you ssh passwordless *back and forth* across *any pair* of computers, using the same user name and UID that you use to launch cpi and other MPI programs? > Clear as mud, hey? > Cristal clear, as any issue with parallel programs. :) I hope it helps, Gus Correa > Thanks for your help. > > Ron > > > <ehddjihc.png> > > > On 2/03/2014 05:20, Gustavo Correa wrote: >> Hi Ron >> >> Please, see my comments inline. >> >> On Mar 1, 2014, at 6:29 AM, Reuti wrote: >> >> >>> Hi, >>> >>> Am 01.03.2014 um 12:18 schrieb Ron Palmer: >>> >>> >>>> Gus and others, >>>> many thanks for your comments and suggestions. I am unsure whether I need fortran included as I am not in control of the software to be run on this linux cluster. I have asked but not yet got a reply... >>>> >>>> However, I did a yum search gfortran and installed the gcc-gfortran.x86_64 (I hope that was the one you referred to). I followed up with the ./configure ... as per your suggestion. That did not work. I then tried with only ./configure --prefix=/my/bin/folder. That worked so I also did make and make install. >>>> >>>> >> Maybe there was a misspell in the configure command line with >> I am not 100% sure about the required gfortran yum RPMs. >> It may also need the corresponding "devel" package, if you want to check, >> but I am not really sure. Maybe you already installed everything you need for >> a functional gfortran. >> >> As far as I can tell from your c2.txt file, configure seems to have recognized gfortran. >> I can't tell for sure if it actually built the MPICH F77 and F90, although chances are that it did. >> However, you may check if you have mpif77 and mpif90 in the MPICH bin directory, >> which would be a good sign. >> >> I would not leave the F77 and F90 interfaces out of the game, >> because sooner or later somebody will come with a Fortran MPI program >> to run on your machine. >> >> >>>> I have a machinefile with three linux computers, all sharing the same mpi binaries. Running hostname on all 20 cores works, so I guess I should conclude that mpi works, right? >>>> >>> Are they using the same binaries /my/bin/folder for a non-interactive startup too, i.e.: >>> >>> $ ssh sargeant which hydra_pmi_proxy >>> >>> returns the same as on the master (maybe ~/.bashrc needs to set the $PATH)? >>> >>> -- Reuti >>> >>> >> Make sure you check Reuti's suggestions. >> >> Did you install MPICH in an NFS shared (across all machines) directory, >> or did you install on each machine in a local directory? >> >> As Reuti pointed out, the environment setup is important. >> If MPICH is not installed in a directory that Linux searches by default, such as /usr or /usr/local, >> you need to add the MPICH bin directory to the PATH, and probably the lib directory to >> LD_LIBRARY_PATH (if built with shared libraries). >> The simplest way is in the .bashrc/.tcshrc file in the user home directory (which I presume >> is shared via NFS across the computers, otherwise you need to edit these files on each >> computer) >> Say: >> >> - bash: >> >> export PATH=${PATH}:/my/mpich/bin >> export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/my/mpich/lib >> >> - tcsh: >> >> setenv PATH ${PATH}:/my/mpich/bin >> setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/my/mpich/lib >> >> Also, you can go a long way troubleshooting these problems if you read the MPICH >> documentation (Installer's Guide, User's Guide, README file) >> >> >> http://www.mpich.org/documentation/guides/ >> >> >> If you really could run mpiexec hostname across all computers, I would guess you >> don't have any security or network related issues (i.e. passwordless ssh works >> across the nodes, there is no firewall blocking, the nodes names are properly >> resolved most likely via /etc/hosts, and so on). >> >> To make sure MPICH works with a simple MPI program, >> compile the cpi.c program in the examples directory (comes with the source code tarball), >> and run it across all nodes. >> >> Something like this: >> >> mpicc -o cpi cpi.c >> mpiexec -machinefile all_machines -np 24 ./cpi >> >> If it runs, then MPICH is alright, and any issues you are having with the other program, >> are not an MPICH issue, but a bug or misconfiguration in the other program. >> >> >> >>>> I then tried the software I am supposed to run on the cluster. It works on all nodes on the master computer, but not if other computers are involved. I have captured the output from such failed attempt to run that software on all available nodes on all available computers in the cluster, see below. I have attached c2.txt and m2.txt (and the output of the failed run) in the attached gzipped file (the failed run in run2.txt and in the image below may not be from the same failed run...). >>>> >>>> >>>> So, is it an issue for the software developers or is it an issue with mpich? >>>> >>>> >> The screenshot you sent MPI ranks 9, 11,12, 14 don't printout their ranks. >> This may or may not indicate a programming error, however combined with the fact that >> the program fails in MPI_Barrier, which is a collective call, suggests it may be a bug in the program. >> One possibility is that the MPI program is wrong, and somehow ranks 9,11,12,14 >> did not call MPI_Barrier (collective calls, as the name says, are collective, >> must be called by all ranks in the "communicator"). >> >> Another line of search would be that in both the screenshot and the run2.txt file >> complain of "Communication error with rank 8" in the MPI_Barrier. >> In each case rank 8 is in a different computer (gainsborough and constable). >> It may be that rank 8 never called MPI_Barrier somehow. >> For instance, MPI collective calls inside "if" conditionals can cause traps like this, >> as the "if" condition may turn out to be true on some ranks and false on others. >> >> Judging from the small amount of output, the program seems to have failed >> very early, which may help search for a possible bug (early in the code). >> >> However, these are just wild guesses. >> >> Is this an in-house program, public domain, or commercial software? >> Anyway, this may be an issue to bring to that code developers. >> >> I hope this helps, >> Gus Correa >> >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problems with running make
by Reuti 02 Mar '14

02 Mar '14

Am 02.03.2014 um 14:01 schrieb Ron Palmer: > I am running centos on two of these, sargeant and constable, and RHEL on the third, gainsborough. > > By runtime libraries, do you mean for the three compilers included, ie gcc, fortran77 and fortran90? Yes. But there should be a package with the runtime libraries, i.e. so that it's not necessary to install the complete compiler suite (I assume in your situation that they are installed, but with different versions). Maybe you can check for older/newer libs in the repository of the distributions. Which libraries are necessary can be listed by issuing: $ ldd cpi on each of the machines and whether they are found. -- Reuti NB: With commercial compilers it's for this reason often allowed in the license to give away the runtime libraries (and only these) with your compiled application, as otherwise your customers couldn't run your applications. > Please use one syllable words, I am the edge of my understanding :-(. > > Thanks > Ron > > > On 2/03/2014 22:55, Reuti wrote: >> Hi, >> >> Am 02.03.2014 um 13:20 schrieb Ron Palmer: >> >>> Gus, Reuti, Rajeev and others, >>> I do have mpif77 and mpif90 in mpi/bin, so I conclude that it has compiled ok. >>> >>> I have checked reuti's suggestions, indicating that the environment is ok. I had added 'PATH=/my/mpi/bin:$PATH' to each of the .bashrc files (for the user I am running on all three) >>> >>> I am using the same username on all three computers - is that ok? >>> >>> I have installed MPI on to a NFS drive, all accessing the same; 'which' confirms. Are there any libraries that needs to be installed elsewhere? do need to run make install as root? >> No. >> >> >>> I have added the 'LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/my/mpi/lib', though made no difference. (should those "{" and "}" be included? I entered it exactly like I have typed) >>> >>> I have mpicc the cpi program as suggested by Gus and tried it, as well as the originally provided cpi (part of the downloaded mpi) which I called cpi_orig, see the screen shot below. It looks like there is an issue with libgfortran.so.3... >> Yep. >> >> Unless you build static applications, the runtime libraries (of the compiler) need to be available on all machines. >> >> There should be package with the runtime libraries in your Linux distribution. >> >> -- Reuti >> >> >>> Perhaps I have not got the various compilers installed correctly? I would appreciate if someone could give me a suggestion on how to properly remove and re-install these (and select the correct ones). >>> >>> By the way, in the runs of cpi below, I reduced the number of cores to be used on the first pc listed in the machinefile, from 8 to 4, in the case there was an issue with the 9th one. As can be seen, it is the first node (or core) on a remote computer where the problems start to be listed which is node 5 in this case... >>> >>> If I run cpi on only one computer, it works. And yes, hostname works even if all computers are included in the machinefile. >>> >>> Clear as mud, hey? >>> Thanks for your help. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problems with running make
by Ron Palmer 02 Mar '14

02 Mar '14

I am running centos on two of these, sargeant and constable, and RHEL on the third, gainsborough. By runtime libraries, do you mean for the three compilers included, ie gcc, fortran77 and fortran90? Please use one syllable words, I am the edge of my understanding :-(. Thanks Ron On 2/03/2014 22:55, Reuti wrote: > Hi, > > Am 02.03.2014 um 13:20 schrieb Ron Palmer: > >> Gus, Reuti, Rajeev and others, >> I do have mpif77 and mpif90 in mpi/bin, so I conclude that it has compiled ok. >> >> I have checked reuti's suggestions, indicating that the environment is ok. I had added 'PATH=/my/mpi/bin:$PATH' to each of the .bashrc files (for the user I am running on all three) >> >> I am using the same username on all three computers - is that ok? >> >> I have installed MPI on to a NFS drive, all accessing the same; 'which' confirms. Are there any libraries that needs to be installed elsewhere? do need to run make install as root? > No. > > >> I have added the 'LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/my/mpi/lib', though made no difference. (should those "{" and "}" be included? I entered it exactly like I have typed) >> >> I have mpicc the cpi program as suggested by Gus and tried it, as well as the originally provided cpi (part of the downloaded mpi) which I called cpi_orig, see the screen shot below. It looks like there is an issue with libgfortran.so.3... > Yep. > > Unless you build static applications, the runtime libraries (of the compiler) need to be available on all machines. > > There should be package with the runtime libraries in your Linux distribution. > > -- Reuti > > >> Perhaps I have not got the various compilers installed correctly? I would appreciate if someone could give me a suggestion on how to properly remove and re-install these (and select the correct ones). >> >> By the way, in the runs of cpi below, I reduced the number of cores to be used on the first pc listed in the machinefile, from 8 to 4, in the case there was an issue with the 9th one. As can be seen, it is the first node (or core) on a remote computer where the problems start to be listed which is node 5 in this case... >> >> If I run cpi on only one computer, it works. And yes, hostname works even if all computers are included in the machinefile. >> >> Clear as mud, hey? >> Thanks for your help.

1 0

Re: [mpich-discuss] Problems with running make
by Reuti 02 Mar '14

02 Mar '14

Hi, Am 02.03.2014 um 13:20 schrieb Ron Palmer: > Gus, Reuti, Rajeev and others, > I do have mpif77 and mpif90 in mpi/bin, so I conclude that it has compiled ok. > > I have checked reuti's suggestions, indicating that the environment is ok. I had added 'PATH=/my/mpi/bin:$PATH' to each of the .bashrc files (for the user I am running on all three) > > I am using the same username on all three computers - is that ok? > > I have installed MPI on to a NFS drive, all accessing the same; 'which' confirms. Are there any libraries that needs to be installed elsewhere? do need to run make install as root? No. > I have added the 'LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/my/mpi/lib', though made no difference. (should those "{" and "}" be included? I entered it exactly like I have typed) > > I have mpicc the cpi program as suggested by Gus and tried it, as well as the originally provided cpi (part of the downloaded mpi) which I called cpi_orig, see the screen shot below. It looks like there is an issue with libgfortran.so.3... Yep. Unless you build static applications, the runtime libraries (of the compiler) need to be available on all machines. There should be package with the runtime libraries in your Linux distribution. -- Reuti > Perhaps I have not got the various compilers installed correctly? I would appreciate if someone could give me a suggestion on how to properly remove and re-install these (and select the correct ones). > > By the way, in the runs of cpi below, I reduced the number of cores to be used on the first pc listed in the machinefile, from 8 to 4, in the case there was an issue with the 9th one. As can be seen, it is the first node (or core) on a remote computer where the problems start to be listed which is node 5 in this case... > > If I run cpi on only one computer, it works. And yes, hostname works even if all computers are included in the machinefile. > > Clear as mud, hey? > Thanks for your help. > > Ron > > > <ehddjihc.png> > > > On 2/03/2014 05:20, Gustavo Correa wrote: >> Hi Ron >> >> Please, see my comments inline. >> >> On Mar 1, 2014, at 6:29 AM, Reuti wrote: >> >> >>> Hi, >>> >>> Am 01.03.2014 um 12:18 schrieb Ron Palmer: >>> >>> >>>> Gus and others, >>>> many thanks for your comments and suggestions. I am unsure whether I need fortran included as I am not in control of the software to be run on this linux cluster. I have asked but not yet got a reply... >>>> >>>> However, I did a yum search gfortran and installed the gcc-gfortran.x86_64 (I hope that was the one you referred to). I followed up with the ./configure ... as per your suggestion. That did not work. I then tried with only ./configure --prefix=/my/bin/folder. That worked so I also did make and make install. >>>> >>>> >> Maybe there was a misspell in the configure command line with >> I am not 100% sure about the required gfortran yum RPMs. >> It may also need the corresponding "devel" package, if you want to check, >> but I am not really sure. Maybe you already installed everything you need for >> a functional gfortran. >> >> As far as I can tell from your c2.txt file, configure seems to have recognized gfortran. >> I can't tell for sure if it actually built the MPICH F77 and F90, although chances are that it did. >> However, you may check if you have mpif77 and mpif90 in the MPICH bin directory, >> which would be a good sign. >> >> I would not leave the F77 and F90 interfaces out of the game, >> because sooner or later somebody will come with a Fortran MPI program >> to run on your machine. >> >> >>>> I have a machinefile with three linux computers, all sharing the same mpi binaries. Running hostname on all 20 cores works, so I guess I should conclude that mpi works, right? >>>> >>> Are they using the same binaries /my/bin/folder for a non-interactive startup too, i.e.: >>> >>> $ ssh sargeant which hydra_pmi_proxy >>> >>> returns the same as on the master (maybe ~/.bashrc needs to set the $PATH)? >>> >>> -- Reuti >>> >>> >> Make sure you check Reuti's suggestions. >> >> Did you install MPICH in an NFS shared (across all machines) directory, >> or did you install on each machine in a local directory? >> >> As Reuti pointed out, the environment setup is important. >> If MPICH is not installed in a directory that Linux searches by default, such as /usr or /usr/local, >> you need to add the MPICH bin directory to the PATH, and probably the lib directory to >> LD_LIBRARY_PATH (if built with shared libraries). >> The simplest way is in the .bashrc/.tcshrc file in the user home directory (which I presume >> is shared via NFS across the computers, otherwise you need to edit these files on each >> computer) >> Say: >> >> - bash: >> >> export PATH=${PATH}:/my/mpich/bin >> export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/my/mpich/lib >> >> - tcsh: >> >> setenv PATH ${PATH}:/my/mpich/bin >> setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/my/mpich/lib >> >> Also, you can go a long way troubleshooting these problems if you read the MPICH >> documentation (Installer's Guide, User's Guide, README file) >> >> >> http://www.mpich.org/documentation/guides/ >> >> >> If you really could run mpiexec hostname across all computers, I would guess you >> don't have any security or network related issues (i.e. passwordless ssh works >> across the nodes, there is no firewall blocking, the nodes names are properly >> resolved most likely via /etc/hosts, and so on). >> >> To make sure MPICH works with a simple MPI program, >> compile the cpi.c program in the examples directory (comes with the source code tarball), >> and run it across all nodes. >> >> Something like this: >> >> mpicc -o cpi cpi.c >> mpiexec -machinefile all_machines -np 24 ./cpi >> >> If it runs, then MPICH is alright, and any issues you are having with the other program, >> are not an MPICH issue, but a bug or misconfiguration in the other program. >> >> >> >>>> I then tried the software I am supposed to run on the cluster. It works on all nodes on the master computer, but not if other computers are involved. I have captured the output from such failed attempt to run that software on all available nodes on all available computers in the cluster, see below. I have attached c2.txt and m2.txt (and the output of the failed run) in the attached gzipped file (the failed run in run2.txt and in the image below may not be from the same failed run...). >>>> >>>> >>>> So, is it an issue for the software developers or is it an issue with mpich? >>>> >>>> >> The screenshot you sent MPI ranks 9, 11,12, 14 don't printout their ranks. >> This may or may not indicate a programming error, however combined with the fact that >> the program fails in MPI_Barrier, which is a collective call, suggests it may be a bug in the program. >> One possibility is that the MPI program is wrong, and somehow ranks 9,11,12,14 >> did not call MPI_Barrier (collective calls, as the name says, are collective, >> must be called by all ranks in the "communicator"). >> >> Another line of search would be that in both the screenshot and the run2.txt file >> complain of "Communication error with rank 8" in the MPI_Barrier. >> In each case rank 8 is in a different computer (gainsborough and constable). >> It may be that rank 8 never called MPI_Barrier somehow. >> For instance, MPI collective calls inside "if" conditionals can cause traps like this, >> as the "if" condition may turn out to be true on some ranks and false on others. >> >> Judging from the small amount of output, the program seems to have failed >> very early, which may help search for a possible bug (early in the code). >> >> However, these are just wild guesses. >> >> Is this an in-house program, public domain, or commercial software? >> Anyway, this may be an issue to bring to that code developers. >> >> I hope this helps, >> Gus Correa >> >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0