I don't know what is this load balancer you are using, but Hydra (the process manager) can interact with several standard job schedulers, such as SLURM and PBS. You can refer to https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager for more information about how to use Hydra. Alternatively, if you want to keep your own scheduler, the only solution that comes to mind is for the scheduler to communicate indirectly with Hydra through a hosts file. The scheduler publishes the list of hosts in a file, say hosts.txt, and this file gets passed to hydra with the -f option: mpiexec -f hosts.txt. It is up to you to synchronize the scheduler and Hydra properly. Halim www.mcs.anl.gov/~aamer On 10/27/16 11:59 PM, [email protected] wrote:
I have a system, that can start and stop machines depending on fleet’s current load. Essentially, it must be the one to decide, which machine will receive a new task, and optionally start acquire a new one from some pool for it.
For example, I can have two hosts of six currently executing something (might be not related to MPICH), and my load balancer machine is aware of it. Other 4 are in low-power state to conserve costs. So when I call mpiexec, I want to tell it to send all the tasks to the load balancer, requesting 2 hosts, so that load balancer then could start two new hosts to handle that request. If the initial hosts would not be busy with some other task, load balancer would send incoming jobs to them, and kept all other four deallocated.
- Victor
*From: *Halim Amer <mailto:[email protected]> *Sent: *Thursday, October 27, 2016 9:18 PM *To: *[email protected] <mailto:[email protected]> *Subject: *Re: [mpich-discuss] Using external load-balancer with mpich
I don't understand what you are trying to do. Can you give an example?
Halim
www.mcs.anl.gov/~aamer
On 10/26/16 5:21 PM, lost wrote:
Can I use an external load balancer with mpiexec by providing a single
hostname in hosts file with, optionally, some large number for host
rank, and putting load balancer listening on that hostname and
forwarding connections to the actual hosts?
I am trying to achieve autoscaling (load balancer tracks liveness of
hosts and spins up new ones on demand).
_______________________________________________
discuss mailing list [email protected]
To manage subscription options or unsubscribe:
_______________________________________________
discuss mailing list [email protected]
To manage subscription options or unsubscribe:
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss