You can specify alternate usernames in a hostfile. See link:

https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Using_Hydra_on_Machines_with_Different_User_Names

You may also want to set HYDRA_LAUNCHER_SSH_ENABLE_WARNINGS=1 to see if SSH will dump out more useful information.

Ken

On 06/14/2017 09:56 AM, Alexander Rast wrote:

Still more conjecture on my part, I think I understand what's going on with the gethostbyname failed message, but have no idea how one would go about fixing it.

The exact message gives 'gethostbyname failed, pi@Burns (errno 1)'. I think this is a case of the underlying implementation not being intelligent enough to strip the username off in order to resolve the host address. In other words, it's looking for a host named 'pi@Burns' rather than a host Burns to which it's going to authenticate as user pi.

If you try to use the hostname without user this fails, because then it tries to authenticate using the current username from the shell that invoked mpiexec (in my case on what will eventually become a 'root host' machine - an ordinary Ubuntu desktop PC) which fails because on the other MPI hosts that username doesn't exist. I have authentication set up using RSA keys which authenticates the local user on the 'root host' desktop PC into the 'working user' on the remote MPI hosts, but it seems the MPI_Comm_spawn function isn't using that authentication path.

What it looks like, is that the MPICH implementation may be such that in order to run MPI_Comm_spawn you have to have an identical username on each host you hope to spawn a process on. I very much *hope* this isn't the case, because this would be exceptionally awkward, it would mean that for each different user wanting to run MPI jobs on the remote cluster they'd have to have their own private user and directory on each node in the cluster.

So 1) is it possible to and 2) how would I, configure the MPI_Comm_spawn command or associated hosts so that you can authenticate as user X on host Y (X@Y) for each spawn host using authorized keys from your user W on machine Z, i.e. on machine Y you have in directory /home/X/.ssh/authorized_keys an entry for W@Z?

On Tue, Jun 13, 2017 at 5:15 PM, Alexander Rast <alex.rast.technical@gmail.com <mailto:alex.rast.technical@gmail.com>> wrote:

    OK, some progress. After more work I determined that the bulk of the
    truly bizarre inconsistencies was down to obnoxious, intrusive
    behaviour in Ubuntu of gnome-keyring-daemon, for example see the
    following post:

    https://askubuntu.com/questions/564821/why-cant-i-interact-with-my-ssh-agent-e-g-ssh-add-d-doesnt-work
    <https://askubuntu.com/questions/564821/why-cant-i-interact-with-my-ssh-agent-e-g-ssh-add-d-doesnt-work>

    which by the way has not been fixed as of 16.04 Ubuntu.

    It seems gnome-keyring-daemon is a particularly badly-behaved
    utility and doesn't help itself with multiple autoload attempts at
    Ubuntu startup. You don't know what ssh agent is actually loaded and
    which RSA keys it has cached. It's also apparently very difficult to
    get rid of, although there are ways. I got it eventually to stop
    loading itself. (Perhaps the MPI community might whinge to Ubuntu
    about this behaviour? Many people have complained about the
    antisocial behaviour of gnome-keyring-daemon but so far Ubuntu's
    response has been: 'we can't see why this should be considered a
    problem. And we have doubts about what you're trying to achieve')

    So now the problem has got to the point where there are 2
    alternative error responses, both occurring at the MPI_Comm_spawn
    command. I've included typical error outputs for both scenarios,
    using the code posted earlier. The 2 errors occur with slightly
    different versions of the configuration file used to spawn the
    processes, which I'm also including. Obviously _2 files go together.

    Any thoughts now on what might be causing either of these 2
    problems? I find the gethostbyname failed messages particularly
    perplexing, since I'm able to ssh into the machines themselves
    without difficulty either by name or IP address.

    On Fri, Jun 9, 2017 at 1:53 PM, Alexander Rast
    <alex.rast.technical@gmail.com

    <mailto:alex.rast.technical@gmail.com>> wrote:

        I've reached a limit of mystification. Attempting to run an MPI
        application using MPI_Comm_spawn from a host is resulting in
        bizarre, inconsistent behaviour of ssh and ssh-askpass.

        What I did is, I created an RSA keypair using ssh-keygen, copied
        the public keys into the ./ssh directories on the machines I'll
        be running MPI on, put them in the authorized_keys file, placed
        all the machines in the known_hosts file on the launcher host
        (which is starting MPI_Comm_spawn), then ran eval ssh-agent and
        added the id_rsa file to the agent on the launcher host.

        You can verify that this part of the system is working because I
        can use ssh directly to access the worker machines that will be
        running the application.

        But when I actually try to run the MPI application, when it gets
        to the spawn, all sorts of wierd and wild stuff happens.
        Sometimes a dialogue (which aggressively grabs focus) comes up
        asking for a password (OpenSSH Authentication). Other times the
        same program has just said that the identity/authenticity of the
        target machine can't be established - do I want to continue? (A
        yes causes authentication to fail). In still other cases, it
        appeared to open the connection but then MPI crashed saying it
        couldn't get the host by name. (yes, every machine has the
        hostnames of every other machine in its hosts file). And in yet
        another case, it seemed to try to run but then crashed saying
        unexpected end-of-file. And so on. There seems to be no rhyme or
        reason to the errors, I can't reproduce anything, each time I
        try it some new and surprising behaviour comes up. What's
        happening? Do I have to do something unusual with machine
        configuration/environment?

        Here are the associated MPI files if anyone wants to look for
        errors. In fact there are probably some errors in the code
        itself, because it's never been able to be debugged (because of
        this wierd behaviour) but I am fairly sure at least the sequence
        through to the spawn command is OK. All help appreciated...

_______________________________________________
discuss mailing list     discuss@mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss