- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Amazon ec2 Windows machine
by Jayesh Krishna 08 Feb '13

08 Feb '13

Hi, How are you running your job (mpiexec command)? Did you try using a machine file to specify the hostnames when running the job? Does the program (cpi) execute correctly on a single ec2 instance? Regards, Jayesh ----- Original Message ----- From: "Nicholas Sgro" <nsgro060(a)gmail.com> To: "Jayesh Krishna" <jayesh(a)mcs.anl.gov> Sent: Thursday, February 7, 2013 9:57:55 PM Subject: Re: [mpich-discuss] Amazon ec2 Windows machine I'm using version 1.4.1p1. I tried the sock channel. It doesn't seem to work either. With sock, I get to the point where I enter the number of intervals, but then it does nothing. Do you know any reason it wouldn't work with ec2 instances? On Thu, Feb 7, 2013 at 4:29 PM, Jayesh Krishna < jayesh(a)mcs.anl.gov > wrote: Hi, Which version of MPICH2 are you using? Did you try the "sock" channel (See if it works)? (PS: We haven't tested MPICH2 on Windows with ec2 instances.) Regards, Jayesh ----- Original Message ----- From: "Nicholas Sgro" < nsgro060(a)gmail.com > To: discuss(a)mpich.org Sent: Thursday, February 7, 2013 11:29:57 AM Subject: [mpich-discuss] Amazon ec2 Windows machine Hi all, I am trying to run the example cpi.exe across 2 amazon ec2 instances running windows. I have different problems depending on the channel I choose. If I try nemesis, I get the following error: Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(392).................: MPID_Init(139)........................: channel initialization failed MPIDI_CH3_Init(38)....................: MPID_nem_init(196)....................: MPIDI_CH3I_Seg_commit(366)............: MPIU_SHMW_Hnd_deserialize(324)........: MPIU_SHMW_Seg_open(863)...............: MPIU_SHMW_Seg_create_attach_templ(763): unable to allocate shared memory - OpenFileMapping The system cannot find the file specified. If I try to use shm, cpi.exe uses 100% of the processors on both machines, but makes no progress and I have to cancel the job. I am attaching logs from smpd from both machines from the runs with nemesis and shm. I don't have any experience with mpich, so I have no idea what the problem is. Any guidance would be appreciated. Thanks _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Amazon ec2 Windows machine
by Jayesh Krishna 07 Feb '13

07 Feb '13

Hi, Which version of MPICH2 are you using? Did you try the "sock" channel (See if it works)? (PS: We haven't tested MPICH2 on Windows with ec2 instances.) Regards, Jayesh ----- Original Message ----- From: "Nicholas Sgro" <nsgro060(a)gmail.com> To: discuss(a)mpich.org Sent: Thursday, February 7, 2013 11:29:57 AM Subject: [mpich-discuss] Amazon ec2 Windows machine Hi all, I am trying to run the example cpi.exe across 2 amazon ec2 instances running windows. I have different problems depending on the channel I choose. If I try nemesis, I get the following error: Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(392).................: MPID_Init(139)........................: channel initialization failed MPIDI_CH3_Init(38)....................: MPID_nem_init(196)....................: MPIDI_CH3I_Seg_commit(366)............: MPIU_SHMW_Hnd_deserialize(324)........: MPIU_SHMW_Seg_open(863)...............: MPIU_SHMW_Seg_create_attach_templ(763): unable to allocate shared memory - OpenFileMapping The system cannot find the file specified. If I try to use shm, cpi.exe uses 100% of the processors on both machines, but makes no progress and I have to cancel the job. I am attaching logs from smpd from both machines from the runs with nemesis and shm. I don't have any experience with mpich, so I have no idea what the problem is. Any guidance would be appreciated. Thanks _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Amazon ec2 Windows machine
by Nicholas Sgro 07 Feb '13

07 Feb '13

Hi all, I am trying to run the example cpi.exe across 2 amazon ec2 instances running windows. I have different problems depending on the channel I choose. If I try nemesis, I get the following error: Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(392).................: MPID_Init(139)........................: channel initialization failed MPIDI_CH3_Init(38)....................: MPID_nem_init(196)....................: MPIDI_CH3I_Seg_commit(366)............: MPIU_SHMW_Hnd_deserialize(324)........: MPIU_SHMW_Seg_open(863)...............: MPIU_SHMW_Seg_create_attach_templ(763): unable to allocate shared memory - OpenFileMapping The system cannot find the file specified. If I try to use shm, cpi.exe uses 100% of the processors on both machines, but makes no progress and I have to cancel the job. I am attaching logs from smpd from both machines from the runs with nemesis and shm. I don't have any experience with mpich, so I have no idea what the problem is. Any guidance would be appreciated. Thanks

1 0

Re: [mpich-discuss] srun env passing
by Biddiscombe, John A. 04 Feb '13

04 Feb '13

Pavan, Dave I hadn't realized that using the with slurm option was preventing the build of mpiexec. I did as you suggested and it works as before. Many thanks for the help. (I also compiled mpich-3.0.2 and it seems to be working fine). JB -----Original Message----- From: discuss-bounces(a)mpich.org [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: 04 February 2013 18:21 To: discuss(a)mpich.org Subject: Re: [mpich-discuss] srun env passing On 02/04/2013 11:16 AM US Central Time, Dave Goodell wrote: > Try rebuilding your MPICH omitting the "--with-pmi=slurm". Instead, > you can use hydra's built-in support for slurm. Some information is > available here: > http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager > > I can't remember exactly what will be auto detected and what won't > (Pavan, can you help me out here?). You might need to set these > environment variables: > > HYDRA_LAUNCHER=slurm HYDRA_RMK=slurm > > Then use "mpiexec" instead of "srun". Everything is auto-detected for slurm. You don't have to set anything. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] shared memory mmap file
by Dustin Sorge 04 Feb '13

04 Feb '13

Fair enough. Thank you very much for the info. Much appreciated, -Dustin On Mon, Feb 4, 2013 at 1:43 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > There shouldn't be any (explicit) file locking going on w.r.t. this file, > at least as far as I know. > > -Dave > > On Feb 4, 2013, at 11:28 AM CST, Dustin Sorge wrote: > > What I was hoping was that I would be able to get some interesting NFS > locking testing by having multiple NFS clients all trying to use this file > while running an MPI job. > > > On Mon, Feb 4, 2013 at 12:08 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > >> On Feb 4, 2013, at 11:05 AM CST, Dustin Sorge wrote: >> >> > I can get MPICH working without it, however I am doing some NFS testing >> and having that file mounted over NFS would be a very interesting use case >> for me. I would like to avoid mounting /tmp over NFS so i wanted to see if >> there was a way to specifying another path for the file. >> >> You probably don't want to be accessing this file directly, and >> especially not over NFS. The only reason a file exists is as a way for all >> processes to map the same memory. IIRC the file should actually be >> unlinked (deleted) as soon as all processes on the node have mmap-ed it. >> >> Can you explain more about what you are actually trying to accomplish? >> >> -Dave >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] shared memory mmap file
by Dave Goodell 04 Feb '13

04 Feb '13

There shouldn't be any (explicit) file locking going on w.r.t. this file, at least as far as I know. -Dave On Feb 4, 2013, at 11:28 AM CST, Dustin Sorge wrote: > What I was hoping was that I would be able to get some interesting NFS locking testing by having multiple NFS clients all trying to use this file while running an MPI job. > > > On Mon, Feb 4, 2013 at 12:08 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > On Feb 4, 2013, at 11:05 AM CST, Dustin Sorge wrote: > > > I can get MPICH working without it, however I am doing some NFS testing and having that file mounted over NFS would be a very interesting use case for me. I would like to avoid mounting /tmp over NFS so i wanted to see if there was a way to specifying another path for the file. > > You probably don't want to be accessing this file directly, and especially not over NFS. The only reason a file exists is as a way for all processes to map the same memory. IIRC the file should actually be unlinked (deleted) as soon as all processes on the node have mmap-ed it. > > Can you explain more about what you are actually trying to accomplish? > > -Dave > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] shared memory mmap file
by Dustin Sorge 04 Feb '13

04 Feb '13

What I was hoping was that I would be able to get some interesting NFS locking testing by having multiple NFS clients all trying to use this file while running an MPI job. On Mon, Feb 4, 2013 at 12:08 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > On Feb 4, 2013, at 11:05 AM CST, Dustin Sorge wrote: > > > I can get MPICH working without it, however I am doing some NFS testing > and having that file mounted over NFS would be a very interesting use case > for me. I would like to avoid mounting /tmp over NFS so i wanted to see if > there was a way to specifying another path for the file. > > You probably don't want to be accessing this file directly, and especially > not over NFS. The only reason a file exists is as a way for all processes > to map the same memory. IIRC the file should actually be unlinked > (deleted) as soon as all processes on the node have mmap-ed it. > > Can you explain more about what you are actually trying to accomplish? > > -Dave > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] srun env passing
by Pavan Balaji 04 Feb '13

04 Feb '13

On 02/04/2013 11:16 AM US Central Time, Dave Goodell wrote: > Try rebuilding your MPICH omitting the "--with-pmi=slurm". Instead, > you can use hydra's built-in support for slurm. Some information is > available here: > http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager > > I can't remember exactly what will be auto detected and what won't > (Pavan, can you help me out here?). You might need to set these > environment variables: > > HYDRA_LAUNCHER=slurm HYDRA_RMK=slurm > > Then use "mpiexec" instead of "srun". Everything is auto-detected for slurm. You don't have to set anything. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] srun env passing
by Dave Goodell 04 Feb '13

04 Feb '13

Try rebuilding your MPICH omitting the "--with-pmi=slurm". Instead, you can use hydra's built-in support for slurm. Some information is available here: http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager I can't remember exactly what will be auto detected and what won't (Pavan, can you help me out here?). You might need to set these environment variables: HYDRA_LAUNCHER=slurm HYDRA_RMK=slurm Then use "mpiexec" instead of "srun". -Dave On Feb 4, 2013, at 9:51 AM CST, Biddiscombe, John A. wrote: > Hello, > I’m working on someone else’s cluster and the mpi is not well setup (i.e. not working). > > I compiled my own mpich2 using > ./configure --with-pmi=slurm --with-pm=no --prefix=/home/biddisco/apps/mpich2-1.5 --enable-shared --disable-static > And a simple hello world executed using srun, works as expected and I get the ranks from all nodes responding correctly. > > When I used mpiexec previously, I ran an MPMD command like this > > mpiexec -n 4 -env DISPLAY :0.0 /home/biddisco/build/pv-38/bin/pvserver -rc -ch=128.178.187.20 -sp=11111 : -n 4 -env DISPLAY :0.1 /home/biddisco/build/pv-38/bin/pvserver -rc -ch=128.178.187.20 -sp=11111 : -n 4 -env DISPLAY :0.2 /home/biddisco/build/pv-38/bin/pvserver -rc -ch=128.178.187.20 -sp=11111 > > I need each of the 3 sets of mpi ranks to have a different DISPLAY environment variable to correctly utilise the GPUs. > > Unfortunately, the mpiexec on the machine does not work with slurm and I’d like to use > srun –n XXX –env XXX /program : –n XXX –env YYY /program ... > etc etc > > but srun doesn’t have an option to pass command line environment vars to the mpi processes. I just can’t find a way of launching an MPMD job using srun with different env vars. > > Is there another way? > > Thanks > > JB > > > -- > John Biddiscombe, email:biddisco @.at.@ cscs.ch > http://www.cscs.ch/ > CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07 > Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] srun env passing
by Pavan Balaji 04 Feb '13

04 Feb '13

Hello, Is there a reason for you to disable the default mpiexec? You could just do: ./configure --prefix=/home/biddisco/apps/mpich2-1.5 --enable-shared --disable-static This will build the default mpiexec in mpich (it'll internally automatically detect slurm and use srun, but with better PMI communication with the processes). Once you have mpiexec, MPMD and other features will work correctly. FWIW, I'd recommend upgrading to mpich-3.0.2. -- Pavan On 02/04/2013 09:51 AM US Central Time, Biddiscombe, John A. wrote: > Hello, > > Iï¿½m working on someone elseï¿½s cluster and the mpi is not well setup > (i.e. not working). > > > > I compiled my own mpich2 using > > ./configure --with-pmi=slurm --with-pm=no > --prefix=/home/biddisco/apps/mpich2-1.5 --enable-shared --disable-static > > And a simple hello world executed using srun, works as expected and I > get the ranks from all nodes responding correctly. > > > > When I used mpiexec previously, I ran an MPMD command like this > > > > mpiexec -n 4 -env DISPLAY :0.0 /home/biddisco/build/pv-38/bin/pvserver > -rc -ch=128.178.187.20 -sp=11111 : -n 4 -env DISPLAY :0.1 > /home/biddisco/build/pv-38/bin/pvserver -rc -ch=128.178.187.20 -sp=11111 > : -n 4 -env DISPLAY :0.2 /home/biddisco/build/pv-38/bin/pvserver -rc > -ch=128.178.187.20 -sp=11111 > > > > I need each of the 3 sets of mpi ranks to have a different DISPLAY > environment variable to correctly utilise the GPUs. > > > > Unfortunately, the mpiexec on the machine does not work with slurm and > Iï¿½d like to use > > srun ï¿½n XXX ï¿½env XXX /program : ï¿½n XXX ï¿½env YYY /program ... > > etc etc > > > > but srun doesnï¿½t have an option to pass command line environment vars to > the mpi processes. I just canï¿½t find a way of launching an MPMD job > using srun with different env vars. > > > > Is there another way? > > > > Thanks > > > > JB > > > > > > -- > > John Biddiscombe, email:biddisco @.at.@ cscs.ch > > http://www.cscs.ch/ > > CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07 > > Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82 > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0