March 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Problem in in executing MPI program in remote machine
by Balaji, Pavan 06 Mar '16

06 Mar '16

Well, first I wanted to make sure that ssh to "node" was indeed the issue, which is why I asked you to try "ssh node" (which, btw, you haven't responded to yet). Anyway, try this: % mpiexec -launcher rsh -f machinefile -n 1 master -- Pavan > On Mar 6, 2016, at 12:45 AM, sanjesh pant <spant3474(a)gmail.com> wrote: > > what is solution for above problem sir > how to execute MPI program in remote node withotut mpiexec in rsh connection > > On Sun, Mar 6, 2016 at 11:45 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > You are using ssh internally when you call "mpiexec". Hence the question. > > -- Pavan > > > On Mar 4, 2016, at 11:28 PM, sanjesh pant <spant3474(a)gmail.com> wrote: > > > > I am using rsh not ssh > > > > On Sat, Mar 5, 2016 at 9:15 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > > > What about "ssh node"? > > > > -- Pavan > > > > > On Mar 4, 2016, at 9:43 PM, sanjesh pant <spant3474(a)gmail.com> wrote: > > > > > > [beowulf@Master mpich-3.2]$ rsh node > > > Last login: Fri Mar 4 11:12:03 from Node > > > [beowulf@Node ~]$ exitrlogin: connection closed. > > > [beowulf@Master mpich-3.2]$ mpiexec -f machinefile -n 1 master > > > Warning: Permanently added the RSA host key for IP address > > > '192.168.11.135' to the list of known hosts. > > > Permission denied, please try again. > > > Permission denied, please try again. > > > Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem in in executing MPI program in remote machine
by sanjesh pant 06 Mar '16

06 Mar '16

what is solution for above problem sir how to execute MPI program in remote node withotut mpiexec in rsh connection On Sun, Mar 6, 2016 at 11:45 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > You are using ssh internally when you call "mpiexec". Hence the question. > > -- Pavan > > > On Mar 4, 2016, at 11:28 PM, sanjesh pant <spant3474(a)gmail.com> wrote: > > > > I am using rsh not ssh > > > > On Sat, Mar 5, 2016 at 9:15 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > > > What about "ssh node"? > > > > -- Pavan > > > > > On Mar 4, 2016, at 9:43 PM, sanjesh pant <spant3474(a)gmail.com> wrote: > > > > > > [beowulf@Master mpich-3.2]$ rsh node > > > Last login: Fri Mar 4 11:12:03 from Node > > > [beowulf@Node ~]$ exitrlogin: connection closed. > > > [beowulf@Master mpich-3.2]$ mpiexec -f machinefile -n 1 master > > > Warning: Permanently added the RSA host key for IP address > > > '192.168.11.135' to the list of known hosts. > > > Permission denied, please try again. > > > Permission denied, please try again. > > > Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem in in executing MPI program in remote machine
by Balaji, Pavan 06 Mar '16

06 Mar '16

You are using ssh internally when you call "mpiexec". Hence the question. -- Pavan > On Mar 4, 2016, at 11:28 PM, sanjesh pant <spant3474(a)gmail.com> wrote: > > I am using rsh not ssh > > On Sat, Mar 5, 2016 at 9:15 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > What about "ssh node"? > > -- Pavan > > > On Mar 4, 2016, at 9:43 PM, sanjesh pant <spant3474(a)gmail.com> wrote: > > > > [beowulf@Master mpich-3.2]$ rsh node > > Last login: Fri Mar 4 11:12:03 from Node > > [beowulf@Node ~]$ exitrlogin: connection closed. > > [beowulf@Master mpich-3.2]$ mpiexec -f machinefile -n 1 master > > Warning: Permanently added the RSA host key for IP address > > '192.168.11.135' to the list of known hosts. > > Permission denied, please try again. > > Permission denied, please try again. > > Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem in in executing MPI program in remote machine
by sanjesh pant 04 Mar '16

04 Mar '16

I am using rsh not ssh On Sat, Mar 5, 2016 at 9:15 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > What about "ssh node"? > > -- Pavan > > > On Mar 4, 2016, at 9:43 PM, sanjesh pant <spant3474(a)gmail.com> wrote: > > > > [beowulf@Master mpich-3.2]$ rsh node > > Last login: Fri Mar 4 11:12:03 from Node > > [beowulf@Node ~]$ exitrlogin: connection closed. > > [beowulf@Master mpich-3.2]$ mpiexec -f machinefile -n 1 master > > Warning: Permanently added the RSA host key for IP address > > '192.168.11.135' to the list of known hosts. > > Permission denied, please try again. > > Permission denied, please try again. > > Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] One-sided semantics
by Balaji, Pavan 04 Mar '16

04 Mar '16

And, of course, I assume you'll eventually fix all the performance shortcomings with respect to doing extra flush/flush_locals or datatype commits inside the for loop. We'll be happy to review the Comex port for you, once it's ready, if you like. -- Pavan > On Mar 4, 2016, at 9:59 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Bruce, > > You are missing an MPI_Barrier on line 81 (after the initialization). Without this, a remote process might update your buffer while you are still initializing. The program works with the barrier. > > -- Pavan > >> On Mar 4, 2016, at 6:22 PM, Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> wrote: >> >> Hi, >> >> I’ve been working on a thin implementation of the COMEX runtime over MPI-3. The COMEX interface has been used by most of the MPI-based runtimes in GA. One of the COMEX tests has processors writing to and then immediately reading from neighboring processes multiple times. The GA semantics are that for multiple consecutive operations between the same pair of processes, the operations are ordered on the remote process in the same order as on the originating process. The test for this frequently fails for the MPI-3 based implementation. I’ve tried testing this independently of GA but the results are confusing. >> >> The implementation I’ve been working on uses three different strategies to implement one-sided communication calls that follow, or are at least close to, the GA communication semantics. The first uses MPI_Put/MPI_Get/MPI_Accumulate and surrounds these calls by and MPI_Lock and MPI_Unlock pair immediately before and after the one-sided communication call. My understanding is that this forces completion both locally and remotely. The second approach calls MPI_Win_lock_all on the MPI window immediately after creation and MPI_Win_unlock_all when the window is destroy so that the window is always in a passive synchronization epoch. The put/get/accumulate calls are implemented with the request-based calls MPI_Rput/MPI_Rget/MPI_Raccumulate and followed immediately by a call to MPI_Wait on the request handle. Again, from my understanding, this should force local completion of the operation but not necessarily remote completion. Finally, the last implementation is to again use the MPI_Win_lock_all to guarantee that a window is in a permanent passive synchronization epoch, use MPI_Put/MPI_Get/MPI_Accumulate to implement put/get/accumulate and use MPI_Win_flush_local to force completion locally. The first implementation should require only a barrier to force synchronization between all processors, the second two include a call to MPI_Win_flush_all in conjunction with a barrier to synchronize the data on all processors. >> >> I’ve written a small test code that implements all three schemes and attached it to this email. It creates a 200x200 array of doubles, fills each array with unique numbers, writes a portion of the array to the next higher rank using put and then reads it back using get (cyclic boundary conditions are used for the first and last ranks). This is repeated 2000 times, with each test using a slightly different set of numbers from the previous test. I’ve done this for all three implementations using both a synchronization between the put and the get and without synchronization. The code has been run on an Infinband cluster using 2 processors on 2 separate SMP nodes. The results I get are that the request-based implementation and the flush_local_all implementation without synchronization work pretty consistently while the tests with synchronization all fail. The lock/unlock implementation also fails both with and without synchronization. Most tests that fail get through at least a few put/get cycles before failing but they don’t do all 2000 iterations. >> >> I’ve also tried this using OpenMPI. In the OpenMPI case, there doesn’t appear to be much of an effect from using synchronization. In addition, the lock/unlock algorithm does not consistently fail, although it fails more frequently than the other two. >> >> Does anyone have a suggestion as to what I’m doing wrong here? From my understanding of the MPI-3 standard, all three implementations should work with synchronization. I’m not completely sure if they should work without synchronization. >> >> Bruce Palmer >> >> <testmpi.c>_______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] One-sided semantics
by Balaji, Pavan 04 Mar '16

04 Mar '16

Bruce, You are missing an MPI_Barrier on line 81 (after the initialization). Without this, a remote process might update your buffer while you are still initializing. The program works with the barrier. -- Pavan > On Mar 4, 2016, at 6:22 PM, Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> wrote: > > Hi, > > I’ve been working on a thin implementation of the COMEX runtime over MPI-3. The COMEX interface has been used by most of the MPI-based runtimes in GA. One of the COMEX tests has processors writing to and then immediately reading from neighboring processes multiple times. The GA semantics are that for multiple consecutive operations between the same pair of processes, the operations are ordered on the remote process in the same order as on the originating process. The test for this frequently fails for the MPI-3 based implementation. I’ve tried testing this independently of GA but the results are confusing. > > The implementation I’ve been working on uses three different strategies to implement one-sided communication calls that follow, or are at least close to, the GA communication semantics. The first uses MPI_Put/MPI_Get/MPI_Accumulate and surrounds these calls by and MPI_Lock and MPI_Unlock pair immediately before and after the one-sided communication call. My understanding is that this forces completion both locally and remotely. The second approach calls MPI_Win_lock_all on the MPI window immediately after creation and MPI_Win_unlock_all when the window is destroy so that the window is always in a passive synchronization epoch. The put/get/accumulate calls are implemented with the request-based calls MPI_Rput/MPI_Rget/MPI_Raccumulate and followed immediately by a call to MPI_Wait on the request handle. Again, from my understanding, this should force local completion of the operation but not necessarily remote completion. Finally, the last implementation is to again use the MPI_Win_lock_all to guarantee that a window is in a permanent passive synchronization epoch, use MPI_Put/MPI_Get/MPI_Accumulate to implement put/get/accumulate and use MPI_Win_flush_local to force completion locally. The first implementation should require only a barrier to force synchronization between all processors, the second two include a call to MPI_Win_flush_all in conjunction with a barrier to synchronize the data on all processors. > > I’ve written a small test code that implements all three schemes and attached it to this email. It creates a 200x200 array of doubles, fills each array with unique numbers, writes a portion of the array to the next higher rank using put and then reads it back using get (cyclic boundary conditions are used for the first and last ranks). This is repeated 2000 times, with each test using a slightly different set of numbers from the previous test. I’ve done this for all three implementations using both a synchronization between the put and the get and without synchronization. The code has been run on an Infinband cluster using 2 processors on 2 separate SMP nodes. The results I get are that the request-based implementation and the flush_local_all implementation without synchronization work pretty consistently while the tests with synchronization all fail. The lock/unlock implementation also fails both with and without synchronization. Most tests that fail get through at least a few put/get cycles before failing but they don’t do all 2000 iterations. > > I’ve also tried this using OpenMPI. In the OpenMPI case, there doesn’t appear to be much of an effect from using synchronization. In addition, the lock/unlock algorithm does not consistently fail, although it fails more frequently than the other two. > > Does anyone have a suggestion as to what I’m doing wrong here? From my understanding of the MPI-3 standard, all three implementations should work with synchronization. I’m not completely sure if they should work without synchronization. > > Bruce Palmer > > <testmpi.c>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem in in executing MPI program in remote machine
by Balaji, Pavan 04 Mar '16

04 Mar '16

What about "ssh node"? -- Pavan > On Mar 4, 2016, at 9:43 PM, sanjesh pant <spant3474(a)gmail.com> wrote: > > [beowulf@Master mpich-3.2]$ rsh node > Last login: Fri Mar 4 11:12:03 from Node > [beowulf@Node ~]$ exitrlogin: connection closed. > [beowulf@Master mpich-3.2]$ mpiexec -f machinefile -n 1 master > Warning: Permanently added the RSA host key for IP address > '192.168.11.135' to the list of known hosts. > Permission denied, please try again. > Permission denied, please try again. > Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem in in executing MPI program in remote machine
by sanjesh pant 04 Mar '16

04 Mar '16

[beowulf@Master mpich-3.2]$ rsh node Last login: Fri Mar 4 11:12:03 from Node [beowulf@Node ~]$ exitrlogin: connection closed. [beowulf@Master mpich-3.2]$ mpiexec -f machinefile -n 1 master Warning: Permanently added the RSA host key for IP address '192.168.11.135' to the list of known hosts. Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

One-sided semantics
by Palmer, Bruce J 04 Mar '16

04 Mar '16

Hi, I've been working on a thin implementation of the COMEX runtime over MPI-3. The COMEX interface has been used by most of the MPI-based runtimes in GA. One of the COMEX tests has processors writing to and then immediately reading from neighboring processes multiple times. The GA semantics are that for multiple consecutive operations between the same pair of processes, the operations are ordered on the remote process in the same order as on the originating process. The test for this frequently fails for the MPI-3 based implementation. I've tried testing this independently of GA but the results are confusing. The implementation I've been working on uses three different strategies to implement one-sided communication calls that follow, or are at least close to, the GA communication semantics. The first uses MPI_Put/MPI_Get/MPI_Accumulate and surrounds these calls by and MPI_Lock and MPI_Unlock pair immediately before and after the one-sided communication call. My understanding is that this forces completion both locally and remotely. The second approach calls MPI_Win_lock_all on the MPI window immediately after creation and MPI_Win_unlock_all when the window is destroy so that the window is always in a passive synchronization epoch. The put/get/accumulate calls are implemented with the request-based calls MPI_Rput/MPI_Rget/MPI_Raccumulate and followed immediately by a call to MPI_Wait on the request handle. Again, from my understanding, this should force local completion of the operation but not necessarily remote completion. Finally, the last implementation is to again use the MPI_Win_lock_all to guarantee that a window is in a permanent passive synchronization epoch, use MPI_Put/MPI_Get/MPI_Accumulate to implement put/get/accumulate and use MPI_Win_flush_local to force completion locally. The first implementation should require only a barrier to force synchronization between all processors, the second two include a call to MPI_Win_flush_all in conjunction with a barrier to synchronize the data on all processors. I've written a small test code that implements all three schemes and attached it to this email. It creates a 200x200 array of doubles, fills each array with unique numbers, writes a portion of the array to the next higher rank using put and then reads it back using get (cyclic boundary conditions are used for the first and last ranks). This is repeated 2000 times, with each test using a slightly different set of numbers from the previous test. I've done this for all three implementations using both a synchronization between the put and the get and without synchronization. The code has been run on an Infinband cluster using 2 processors on 2 separate SMP nodes. The results I get are that the request-based implementation and the flush_local_all implementation without synchronization work pretty consistently while the tests with synchronization all fail. The lock/unlock implementation also fails both with and without synchronization. Most tests that fail get through at least a few put/get cycles before failing but they don't do all 2000 iterations. I've also tried this using OpenMPI. In the OpenMPI case, there doesn't appear to be much of an effect from using synchronization. In addition, the lock/unlock algorithm does not consistently fail, although it fails more frequently than the other two. Does anyone have a suggestion as to what I'm doing wrong here? From my understanding of the MPI-3 standard, all three implementations should work with synchronization. I'm not completely sure if they should work without synchronization. Bruce Palmer _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem in in executing MPI program in remote machine
by sanjesh pant 04 Mar '16

04 Mar '16

Sir i am using rsh command for remote connection and it is working On Fri, Mar 4, 2016 at 7:22 PM, Halim Amer <aamer(a)anl.gov> wrote: > It's a problem in your ssh settings not MPICH. > Make sure communication between all your nodes is well setup before > running an MPI program. > Some more hints are given in our FAQ page > https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI… > > --Halim > www.mcs.anl.gov/~aamer > > > On 3/3/16 10:40 PM, sanjesh pant wrote: > >> Hi Sir, >> As per your suggesstion got from mail , I used it to execute >> program but still there is a problem . >> Now I am having permission problem. >> Problem : Permission denied (publickey, gssapi-keyex, gssapi-with-mic). >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0