October 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Is there a way to set timeout to mpi process launch?
by Pranav Ladkat 31 Oct '16

31 Oct '16

Specifying MPIEXEC_TIMEOUT would not be possible since the execution times would vary depending on the different job. Is there any solution available within mpich? Thanks, Pranav On Thu, Oct 27, 2016 at 10:15 PM, Halim Amer <aamer(a)anl.gov> wrote: > You can try setting MPIEXEC_TIMEOUT=<timeout value in seconds> to force > the job to abort after running for the specified period. This is for the > whole execution though, not just for the process launching step. > > Halim > www.mcs.anl.gov/~aamer > > > On 10/26/16 6:30 PM, Pranav Ladkat wrote: > >> Hi, >> >> When I run mpi program on multiple hosts, if executable fails to start >> on any of the host (due to missing library etc. type of reasons), other >> hosts just keep waiting for the process to come up. The program just >> hangs forever. Is there any way to set a timeout in such cases such that >> MPI should abort if not all processes were launched in a given timeout >> period? >> >> Thanks, >> Pranav >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Using external load-balancer with mpich
by Halim Amer 28 Oct '16

28 Oct '16

I don't know what is this load balancer you are using, but Hydra (the process manager) can interact with several standard job schedulers, such as SLURM and PBS. You can refer to https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager for more information about how to use Hydra. Alternatively, if you want to keep your own scheduler, the only solution that comes to mind is for the scheduler to communicate indirectly with Hydra through a hosts file. The scheduler publishes the list of hosts in a file, say hosts.txt, and this file gets passed to hydra with the -f option: mpiexec -f hosts.txt. It is up to you to synchronize the scheduler and Hydra properly. Halim www.mcs.anl.gov/~aamer On 10/27/16 11:59 PM, lostfreeman(a)gmail.com wrote: > I have a system, that can start and stop machines depending on fleet’s > current load. Essentially, it must be the one to decide, which machine > will receive a new task, and optionally start acquire a new one from > some pool for it. > > > > For example, I can have two hosts of six currently executing something > (might be not related to MPICH), and my load balancer machine is aware > of it. Other 4 are in low-power state to conserve costs. So when I call > mpiexec, I want to tell it to send all the tasks to the load balancer, > requesting 2 hosts, so that load balancer then could start two new hosts > to handle that request. If the initial hosts would not be busy with some > other task, load balancer would send incoming jobs to them, and kept all > other four deallocated. > > > > - Victor > > > > *From: *Halim Amer <mailto:[email protected]> > *Sent: *Thursday, October 27, 2016 9:18 PM > *To: *discuss(a)mpich.org <mailto:[email protected]> > *Subject: *Re: [mpich-discuss] Using external load-balancer with mpich > > > > I don't understand what you are trying to do. Can you give an example? > > > > Halim > > www.mcs.anl.gov/~aamer > > > > On 10/26/16 5:21 PM, lost wrote: > >> Can I use an external load balancer with mpiexec by providing a single > >> hostname in hosts file with, optionally, some large number for host > >> rank, and putting load balancer listening on that hostname and > >> forwarding connections to the actual hosts? > >> > >> I am trying to achieve autoscaling (load balancer tracks liveness of > >> hosts and spins up new ones on demand). > >> > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Is there a way to set timeout to mpi process launch?
by Halim Amer 28 Oct '16

28 Oct '16

You can try setting MPIEXEC_TIMEOUT=<timeout value in seconds> to force the job to abort after running for the specified period. This is for the whole execution though, not just for the process launching step. Halim www.mcs.anl.gov/~aamer On 10/26/16 6:30 PM, Pranav Ladkat wrote: > Hi, > > When I run mpi program on multiple hosts, if executable fails to start > on any of the host (due to missing library etc. type of reasons), other > hosts just keep waiting for the process to come up. The program just > hangs forever. Is there any way to set a timeout in such cases such that > MPI should abort if not all processes were launched in a given timeout > period? > > Thanks, > Pranav > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Using external load-balancer with mpich
by lostfreeman＠gmail.com 27 Oct '16

27 Oct '16

I have a system, that can start and stop machines depending on fleet’s current load. Essentially, it must be the one to decide, which machine will receive a new task, and optionally start acquire a new one from some pool for it. For example, I can have two hosts of six currently executing something (might be not related to MPICH), and my load balancer machine is aware of it. Other 4 are in low-power state to conserve costs. So when I call mpiexec, I want to tell it to send all the tasks to the load balancer, requesting 2 hosts, so that load balancer then could start two new hosts to handle that request. If the initial hosts would not be busy with some other task, load balancer would send incoming jobs to them, and kept all other four deallocated. - Victor From: Halim Amer Sent: Thursday, October 27, 2016 9:18 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Using external load-balancer with mpich I don't understand what you are trying to do. Can you give an example? Halim www.mcs.anl.gov/~aamer On 10/26/16 5:21 PM, lost wrote: > Can I use an external load balancer with mpiexec by providing a single > hostname in hosts file with, optionally, some large number for host > rank, and putting load balancer listening on that hostname and > forwarding connections to the actual hosts? > > I am trying to achieve autoscaling (load balancer tracks liveness of > hosts and spins up new ones on demand). > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Using external load-balancer with mpich
by Halim Amer 27 Oct '16

27 Oct '16

I don't understand what you are trying to do. Can you give an example? Halim www.mcs.anl.gov/~aamer On 10/26/16 5:21 PM, lost wrote: > Can I use an external load balancer with mpiexec by providing a single > hostname in hosts file with, optionally, some large number for host > rank, and putting load balancer listening on that hostname and > forwarding connections to the actual hosts? > > I am trying to achieve autoscaling (load balancer tracks liveness of > hosts and spins up new ones on demand). > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Running a program on multiple computers
by Kenneth Raffenetti 27 Oct '16

27 Oct '16

Hi, Apologies for the delay in response. My comments inline below. On 10/15/2016 07:40 PM, Mahdi, Sam wrote: > Hello everyone, > > I am attempting to run a single program on 32 cores split across 4 > computers (So each computer has 8 cores). I am attempting to use mpich > for this. I currently am just testing on 2 computers, I have the program > installed on both, as well as mpich installed on both. I have created a > register key and can login in using ssh into the other computer without > a password. I have come across 2 problems. One, when I attempt to > connect using the mpirun -np 3 --host a (the IP of the computer I am > attempting to connect to) hostname > I recieve the error > unable to connect from "localhost.localdomain" to "localhost.localdomain" > > This is indicating my computers "localhost.localdomain" is attempting to > connect to another "localhost.localdomain". How can I change this so > that it connects via my IP to the other computers IP? > > Secondly, I attempted to use a host file instead using the hydra process > wiki. I created a hosts file with just the IP of the computer I am > attempting to connect to. When I type in the command mpiexec -f hosts -n > 4 ./applic > > I get this error > [mpiexec(a)localhost.localdomain] HYDU_parse_hostfile > (./utils/args/args.c:323): unable to open host file: hosts > > along with other errors of unable to parse hostfile, match handler etc. > I assume this is all due to it being unable to read the host file. Is > there any specific place I should save my hosts file? I have it saved > directly on my Desktop. I have attempted to indicate the full path where > it is located, but I still get the same error. There is no required location for the hosts file. If you are specifying full path and there are still issues, it may be a formatting issue. Can you paste or attach the contents of your hosts file so we can confirm the format is good? > > For the first problem, I have read that I need to change /etc/hosts > manually by using the sudo command to manually enter the IP of the > computer I am attempting to connect to in the /etc/hosts file. I assume > the computer is attempting to connect to itself (set up the program > first on its own core, then send it to another, hence attempting to > start it on localhost.localdomain). > > For the second problem, I have attempted to add run the command > mpirun --host my computer IP, the other computer IP ./program This format should be okay for your purposes. What happens if you try: mpirun --host my computer IP, the other computer IP /bin/hostname If the hostnames of each host are echoed to the command-line, then job launch is successful and the issues is during connection setup during MPI_Init. Ken _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Is there a way to set timeout to mpi process launch?
by Pranav Ladkat 26 Oct '16

26 Oct '16

Hi, When I run mpi program on multiple hosts, if executable fails to start on any of the host (due to missing library etc. type of reasons), other hosts just keep waiting for the process to come up. The program just hangs forever. Is there any way to set a timeout in such cases such that MPI should abort if not all processes were launched in a given timeout period? Thanks, Pranav _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Using external load-balancer with mpich
by lost 26 Oct '16

26 Oct '16

Can I use an external load balancer with mpiexec by providing a single hostname in hosts file with, optionally, some large number for host rank, and putting load balancer listening on that hostname and forwarding connections to the actual hosts? I am trying to achieve autoscaling (load balancer tracks liveness of hosts and spins up new ones on demand). _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Running a program on multiple computers
by Mahdi, Sam 15 Oct '16

15 Oct '16

Hello everyone, I am attempting to run a single program on 32 cores split across 4 computers (So each computer has 8 cores). I am attempting to use mpich for this. I currently am just testing on 2 computers, I have the program installed on both, as well as mpich installed on both. I have created a register key and can login in using ssh into the other computer without a password. I have come across 2 problems. One, when I attempt to connect using the mpirun -np 3 --host a (the IP of the computer I am attempting to connect to) hostname I recieve the error unable to connect from "localhost.localdomain" to "localhost.localdomain" This is indicating my computers "localhost.localdomain" is attempting to connect to another "localhost.localdomain". How can I change this so that it connects via my IP to the other computers IP? Secondly, I attempted to use a host file instead using the hydra process wiki. I created a hosts file with just the IP of the computer I am attempting to connect to. When I type in the command mpiexec -f hosts -n 4 ./applic I get this error [mpiexec(a)localhost.localdomain] HYDU_parse_hostfile (./utils/args/args.c:323): unable to open host file: hosts along with other errors of unable to parse hostfile, match handler etc. I assume this is all due to it being unable to read the host file. Is there any specific place I should save my hosts file? I have it saved directly on my Desktop. I have attempted to indicate the full path where it is located, but I still get the same error. For the first problem, I have read that I need to change /etc/hosts manually by using the sudo command to manually enter the IP of the computer I am attempting to connect to in the /etc/hosts file. I assume the computer is attempting to connect to itself (set up the program first on its own core, then send it to another, hence attempting to start it on localhost.localdomain). For the second problem, I have attempted to add run the command mpirun --host my computer IP, the other computer IP ./program This now gives the error that localhost.localdomain cannot connect to my computers IP (it cannot create a connection). Sincerely, Sam Sincerely, Sam _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Errors building MPICH on Windows 10 Ubuntu subsystem
by Damian Rouson 11 Oct '16

11 Oct '16

> On Oct 11, 2016, at 10:22 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > > On 10/11/2016 10:05 AM, Kenneth Raffenetti wrote: >> On 10/11/2016 09:45 AM, Damian Rouson wrote: >>>> > Also, if you have Ubuntu, can you simply apt-get install mpich? >>> On this system, apt-get installs GCC 4.8. My goal is to support users >>> of OpenCoarrays, which requires at least GCC 5.1.0 and preferably GCC >>> 6.1.0. Is there a Debian dpkg file that I could use to install MPICH? >> >> Yes, try 'apt-get install mpich'. > > I think he's saying "apt-get install mpich" brings in gcc-4.8 as a dependency. Actually, “sudo apt-get install mpich” only installs mpirun and mpiexec, but not mpif90, which is required for OpenCoarrays. This seems very strange, but even if it apt-get installed mpif90, I would need for it to be wrapping a newer version of gfortran than is currently installed by apt-get on Windows 10 Subsystem for Linux. > > Perhaps "apt-get source mpich" or maybe with the "-b" flag for bulid-dependencies, then re-building the debian package will get you what you need? > “sudo apt-get source mpich” yields “E: You must put some ‘source’ URIs in your sources.list.” Adding -b before or after “source” generates the same result. Damian _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0