Re: [mpich-discuss] MPICH 2 script to MPICH 3

19 Jan 2015

      FYI, mpd uses ssh also, so that shouldn't be an issue.  I wouldn't recommend the manual launch.  It's only meant for tools developers who build stuff on top of Hydra.

AFAICT, you can remove most of the mpd management code in your script and simply pass the host list to mpiexec in Hydra:

http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager

If Condor stores the host list allocated for your job somewhere, try passing it to Hydra with the "-f" option to mpiexec.

  -- Pavan
...
On Jan 19, 2015, at 10:04 AM, Kenneth Raffenetti <[email protected]> wrote:
Hi Livan,
It should be possible to convert your script to use Hydra. Hydra has 2 main components. The first is the main hydra executable - mpiexec. The other is the hydra_pmi_proxy.
In a normal ssh launched scenario, mpiexec will launch proxies on all nodes being used for the job, and those proxies will launch the mpi processes.
If ssh is not possible in your setup, you should be able to utilize the hydra manual laucher (mpiexec -launcher manual). When you run mpiexec with that parameter, it will output the proxy launch commands that can be used to connect to it. It will then wait for all the proxies to connect. Here's an example:
raffenet@doom:mpich/ $ mpiexec -launcher manual -n 3 -hosts a,b,c /bin/hostname
HYDRA_LAUNCH: /sandbox/mpich/i/bin/hydra_pmi_proxy --control-port doom:36311 --rmk user --launcher manual --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0
HYDRA_LAUNCH: /sandbox/mpich/i/bin/hydra_pmi_proxy --control-port doom:36311 --rmk user --launcher manual --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 1
HYDRA_LAUNCH: /sandbox/mpich/i/bin/hydra_pmi_proxy --control-port doom:36311 --rmk user --launcher manual --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 2
HYDRA_LAUNCH_END
Let me know if you have other questions.
Ken
On 01/19/2015 05:39 AM, Livan Valladares wrote:
...
Hello,
I have been using Htcondor with MPICH 2 but I update MPICH 2 to MPICH 3 and now the MPD process manager has been deprecated and Hydra is de default process manager.
The script I was using use some commands like mpdtrace, mpdallexit but they are not supported anymore.
Here is my script, is it any possibility to change this script code to be able to work with Hydra?
Thank you very much,
Livan Valladares Martell
Script:
#!/bin/sh
##**************************************************************
##
## Copyright (C) 1990-2014, Condor Team, Computer Sciences Department,
## University of Wisconsin-Madison, WI.
##
## Licensed under the Apache License, Version 2.0 (the "License"); you
## may not use this file except in compliance with the License.  You may
## obtain a copy of the License at
##
##    http://www.apache.org/licenses/LICENSE-2.0
##
## Unless required by applicable law or agreed to in writing, software
## distributed under the License is distributed on an "AS IS" BASIS,
## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
## See the License for the specific language governing permissions and
## limitations under the License.
##
##**************************************************************
# Set this to the bin directory of MPICH installation
MPDIR=/opt/local/bin
PATH=$MPDIR:.:$PATH
export PATH
_CONDOR_PROCNO=$_CONDOR_PROCNO
_CONDOR_NPROCS=$_CONDOR_NPROCS
# Remove the contact file, so if we are held and released
# it can be recreated anew
rm -f $CONDOR_CONTACT_FILE
PATH=`condor_config_val libexec`/:$PATH
# mpd needs a conf file, and it must be
# permissions 0700
mkdir tmp
MPD_CONF_FILE=`pwd`/tmp/mpd_conf_file
export MPD_CONF_FILE
ulimit -c 0
# If you have a shared file system, maybe you
# want to put the mpd.conf file in your home
# directory
echo "password=somepassword" > $MPD_CONF_FILE
chmod 0700 $MPD_CONF_FILE
# If on the head node, start mpd, get the port and host,
# and condor_chirp it back into the ClassAd
# so the non-head nodes can find the head node.
if [ $_CONDOR_PROCNO -eq 0 ]
then
  mpd > mpd.out.$_CONDOR_PROCNO 2>&1 &
  sleep 1
  host=`mpdtrace -l | sed 1q | tr '_' ' ' | awk '{print $1}'`
  port=`mpdtrace -l | sed 1q | tr '_' ' ' | awk '{print $2}'`
condor_chirp set_job_attr MPICH_PORT $port
  condor_chirp set_job_attr MPICH_HOST \"$host\"
num_hosts=1
  retries=0
  while [ $num_hosts -ne $_CONDOR_NPROCS ]
  do
      num_hosts=`mpdtrace | wc -l`
      sleep 2
      retries=`expr $retries + 1`
      if [ $retries -gt 100 ]
      then
      	echo "Too many retries, could not start all $_CONDOR_NPROCS nodes, only started $num_hosts, giving up.  Here are the hosts I could start "
      	mpdtrace
      	exit 1
      fi
  done
## run the actual mpi job, which was the command line argument
  ## to the invocation of this shell script
  mpiexec -n $_CONDOR_NPROCS $@
  e=$?
mpdallexit
  sleep 20
  echo $e
else
  # If NOT the head node, acquire the host and port of
  # the head node
  retries=0
  host=UNDEFINED
  while [ $host == "UNDEFINED" ]
  do
      host=`condor_chirp get_job_attr MPICH_HOST`
      sleep 2
      retries=`expr $retries + 1`
      if [ $retries -gt 100 ]; then
                    echo "Too many retries, could not get mpd host from condor_chirp, giving up."
                    exit 1
                fi
  done
port=`condor_chirp get_job_attr MPICH_PORT`
  host=`echo $host | tr -d '"'`
  mpd --host=$host --port=$port > mpd.out.$_CONDOR_PROCNO 2>&1
fi
_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss
--
Pavan Balaji  ✉️
http://www.mcs.anl.gov/~balaji

_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss

Re: [mpich-discuss] MPICH 2 script to MPICH 3

Balaji, Pavan