April 2014 - discuss - mailman3.cels.anl.gov

Reference paper for ROMIO
by Matthieu Dorier 10 Apr '14

10 Apr '14

Hi, For citation, what is the reference paper for ROMIO? Thanks, Matthieu Dorier PhD student at ENS Rennes http://people.irisa.fr/Matthieu.Dorier

1 0

Re: [mpich-discuss] my mpi code goes slow after some time
by Lu, Huiwei 10 Apr '14

10 Apr '14

How slow is that? What’s the version of your MPI? How many processes you use? If only 0.3% RAM was used for each core, it was unlikely the code went slow because of running out of RAM. Can you give us a minimum working example of your code so that we can reproduce your problem? — Huiwei Lu Postdoc Appointee Mathematics and Computer Science Division, Argonne National Laboratory http://www.mcs.anl.gov/~huiweilu/ On Apr 10, 2014, at 7:57 AM, harsh soni <harshiisc(a)gmail.com> wrote: > Hi, > I am running a mpi code on cluster. It goes slow after some time. I don't allocate any array in main for loop, every array is allocated initially. Each core is showing 0.3 to 0.5 % RAM usage. doe sit go slow due to the RAM. I use fortran language. > > > Thanks > > With best regards > Harsh Soni > Ph.D. student, > IISc, Bengaluru (India) > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] my mpi code goes slow after some time
by harsh soni 10 Apr '14

10 Apr '14

> > Hi, > I am running a mpi code on cluster. It goes slow after some time. I don't > allocate any array in main for loop, every array is allocated initially. > Each core is showing 0.3 to 0.5 % RAM usage. doe sit go slow due to the > RAM. I use fortran language. > > > Thanks > > With best regards > Harsh Soni <http://www.physics.iisc.ernet.in/%7Eharshsoni/> > Ph.D. student, > IISc <http://www.iisc.ernet.in/>, Bengaluru (India) > >

1 0

Re: [mpich-discuss] MPIExec.hydra allocates too many CPU's on SLURM
by Balaji, Pavan 10 Apr '14

10 Apr '14

Ruben, If you want Hydra to use SLURM’s allocation, you should not provide your own allocation on nodes on the mpiexec command-line. You could do something like this: % mpiexec.hydra -n 1 ./master : -n 7 ./slave Hydra will automatically detect the slurm environment and pick the hosts from there. You could also do: % mpiexec.hydra -bootstrap slurm -rmk slurm -n 1 ./master : -n 7 ./slave but that’s redundant, since the “rmk” and “launcher” are automatically detected. “bootstrap” is deprecated at this point. Please don’t use it. You could use “launcher” instead, but that’s autodetected anyway. The following is *not* what you want to do: % mpiexec.hydra -hosts node2-a,node2-b -bootstrap slurm -rmk slurm -n 1 ./master : -n 7 ./slave The “-hosts” option overrides the resource allocation from the “rmk”. So you are telling hydra to ignore SLURM’s resource allocation and use what you have provided. The model is available for expert users who want to explicitly manage their own resources. — Pavan On Apr 10, 2014, at 6:49 AM, Ruben Faelens <faelens(a)kth.se> wrote: > Hello list, > > I have been messing around with SLURM and MPICH2 for about a week now, in order to run NONMEM on my cluster. > > I start my job using the following command line: > > mpiexec.hydra -hosts node2-a,node2-b -bootstrap slurm -rmk slurm -n 1 ./master : -n 7 ./slave > > Using pstree, I can see that this launches the following command: > > srun -N 4 -n 4 /usr/local/bin/hydra_pmi_proxy --control-port node2-head:38860 --rmk slurm --launcher slurm --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id -1 > > Apparently, Hydra launches one single 'hydra_pmi_proxy' per node, after which the hydra_pmi_proxy launches the other processes. This completely screws up allocation rules in SLURM. > > Is this normal behaviour? I would rather have mpiexec.hydra allocate the right number of resources on SLURM, instead of seemingly deciding on its own which hosts to use. > > Kind regards, > Ruben FAELENS > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPIExec.hydra allocates too many CPU's on SLURM
by Kenneth Raffenetti 10 Apr '14

10 Apr '14

What version of mpich are you using? Why are you specifying -hosts in your mpiexec command? Are you allocating resources in some other way prior to launching your job? You are correct about the normal behavior of hydra. It launches a proxies on all nodes which then launch the MPI processes. On 04/10/2014 06:49 AM, Ruben Faelens wrote: > Hello list, > > I have been messing around with SLURM and MPICH2 for about a week now, > in order to run NONMEM on my cluster. > > I start my job using the following command line: > > mpiexec.hydra -hosts node2-a,node2-b -bootstrap slurm -rmk slurm -n 1 > ./master : -n 7 ./slave > > Using pstree, I can see that this launches the following command: > > srun -N 4 -n 4 /usr/local/bin/hydra_pmi_proxy --control-port > node2-head:38860 --rmk slurm --launcher slurm --demux poll --pgid 0 > --retries 10 --usize -2 --proxy-id -1 > > Apparently, Hydra launches one single 'hydra_pmi_proxy' per node, after > which the hydra_pmi_proxy launches the other processes. This completely > screws up allocation rules in SLURM. > > Is this normal behaviour? I would rather have mpiexec.hydra allocate the > right number of resources on SLURM, instead of seemingly deciding on its > own which hosts to use. > > Kind regards, > Ruben FAELENS > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

MPIExec.hydra allocates too many CPU's on SLURM
by Ruben Faelens 10 Apr '14

10 Apr '14

Hello list, I have been messing around with SLURM and MPICH2 for about a week now, in order to run NONMEM on my cluster. I start my job using the following command line: > mpiexec.hydra -hosts node2-a,node2-b -bootstrap slurm -rmk slurm -n 1 ./master : -n 7 ./slave Using pstree, I can see that this launches the following command: > srun -N 4 -n 4 /usr/local/bin/hydra_pmi_proxy --control-port node2-head:38860 --rmk slurm --launcher slurm --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id -1 Apparently, Hydra launches one single 'hydra_pmi_proxy' per node, after which the hydra_pmi_proxy launches the other processes. This completely screws up allocation rules in SLURM. Is this normal behaviour? I would rather have mpiexec.hydra allocate the right number of resources on SLURM, instead of seemingly deciding on its own which hosts to use. Kind regards, Ruben FAELENS

1 0

Re: [mpich-discuss] mpiexec
by Balaji, Pavan 09 Apr '14

09 Apr '14

On Apr 9, 2014, at 12:53 PM, Gus Correa <gus(a)ldeo.columbia.edu> wrote: > For the benefit of new users, > this information could be more visible, though. > Perhaps an entry in the User's Guide or the FAQ. Good idea. I’ll add it to the user guide. FAQ is probably not appropriate for this, unless, of course, we get this question frequently enough. :-) — Pavan

1 0

MPI-IO bug
by Wei-keng Liao 09 Apr '14

09 Apr '14

(This bug is probably caused by my patch long ago.) Attached is a program extracted from an application that can reproduce the problem observed from a large run. The problem is when defining a filetype using MPI_Type_indexed and the first few elements of argument blocklens[] are zeros, a collective write will miss writing some data. The test program first fills a file with 9 integers with values all -999. It then defines a filetype and writes to the file in parallel with user buffers with value all 1s. Lastly, the file is read back and checked for contents. Command used to compile and run the test program: mpicc -g -o bug_indexed_io bug_indexed_io.c mpiexec -n 2 bug_indexed_io Stdout: 0: Error: unexpected varlue at buf[7] == -999 The patch below can fix this problem. Hope it does not break other tests. Index: adio/common/ad_read_coll.c @@ -368,13 +368,16 @@ #endif if (file_ptr_type == ADIO_INDIVIDUAL) { /* Wei-keng reworked type processing to be a bit more efficient */ + for (i=0; i<flat_file->count; i++) /* skip blocklens[] == 0 */ + if (flat_file->blocklens[i] > 0) break; + offset = fd->fp_ind - disp; - n_filetypes = (offset - flat_file->indices[0]) / filetype_extent; + n_filetypes = (offset - flat_file->indices[i]) / filetype_extent; offset -= (ADIO_Offset)n_filetypes * filetype_extent; /* now offset is local to this extent */ /* find the block where offset is located, skip blocklens[i]==0 */ - for (i=0; i<flat_file->count; i++) { + for (; i<flat_file->count; i++) { ADIO_Offset dist; if (flat_file->blocklens[i] == 0) continue; dist = flat_file->indices[i] + flat_file->blocklens[i] - offset; Wei-keng

1 0

Re: [mpich-discuss] mpiexec
by Gus Correa 09 Apr '14

09 Apr '14

I guess item (h) in the MPICH README file may help you: https://www.mpich.org/static/downloads/3.1/mpich-3.1-README.txt For the benefit of new users, this information could be more visible, though. Perhaps an entry in the User's Guide or the FAQ. My two cents, Gus Correa On 04/09/2014 07:04 AM, fereshteh komijani wrote: > as i know before machine file you nead ./ as bellow: > mpiexec -n 5 -f ./machinefile ./examples/cpi > > > On Wed, Apr 9, 2014 at 11:27 AM, Mojtaba S. Moghani > <mshokrollahi(a)ut.ac.ir <mailto:[email protected]>> wrote: > > __ > > How could I define a list of machines? to issue the mpiexec command > in this step : mpiexec -n 5 -f machinefile ./examples/cpi > > could you tell me after installing step, what should I exactly do then > > thanks, > > -- > Mojtaba S. Moghani > Faculty of Environment > University of Tehran > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > ***Angel*** > *__* > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] lldb and mpich issue:
by Dave Goodell (dgoodell) 09 Apr '14

09 Apr '14

Sounds like something worth reporting upstream to LLVM in addition to an MPICH-specific workaround. -Dave On Apr 8, 2014, at 8:47 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > Drew, > > Thanks for reporting this issue. I’m able to reproduce this on a mac with lldb. > > On some initial digging it looks like lldb is messing with UNIX sockets created between mpiexec’s proxy and the MPI process. lldb should be seeing that socket, but it doesn’t seem to hand it off to the application process. That’s nasty. > > I’ve created a ticket for this: > > https://trac.mpich.org/projects/mpich/ticket/2063 > > The simplest solution will be to give an option in mpiexec to avoid the UNIX sockets path completely and use TCP/IP sockets. We don’t want to do this by default because: (1) it requires the system to allow internal TCP connections and (2) UNIX sockets can be a bit faster in some cases. We’ll try to get this into mpich soon. > > Thanks, > > — Pavan > > On Apr 8, 2014, at 8:19 PM, Drew Lewis <drew90(a)vt.edu> wrote: > >> Dear MPICH discuss group, >> >> Sorry if this is a duplicate, I tried to send it before being subscribed, and am not sure if it ever made it to the list. >> >> I am encountering an issue when trying to use lldb with both mpich 3.1 and 3.0.4. >> >> The command that generates the error is >> >> computer:tests ME$ mpirun -np 1 lldb ./a.out >> Current executable set to './a.out' (x86_64). >> (lldb) run >> run >> Process 67202 launched: './a.out' (x86_64) >> [cli_0]: write_line error; fd=6 buf=:cmd=init pmi_version=1 pmi_subversion=1 >> : >> system msg for write_line failure : Bad file descriptor >> [cli_0]: Unable to write to PMI_fd >> [cli_0]: write_line error; fd=6 buf=:cmd=get_appnum >> : >> system msg for write_line failure : Bad file descriptor >> Fatal error in MPI_Init: Other MPI error, error stack: >> MPIR_Init_thread(467): >> MPID_Init(140).......: channel initialization failed >> MPID_Init(422).......: PMI_Get_appnum returned -1 >> Process 67202 exited with status = 1 (0x00000001) >> >> mpich 3.1 was configured with >> $SRC_DIR/configure \ >> --prefix=$INSTALL_ROOT/$BUILD_DIR \ >> --disable-cxx \ >> --disable-f77 \ >> --disable-fc \ >> CC=clang \ >> CXX=clang++ \ >> >> The version of lldb is >> lldb-310.2.36 >> >> The minimal example that shows the problem is >> #include <mpi.h> >> >> int main(int argc, char**argv){ >> MPI_Init(&argc, &argv); >> MPI_Finalize(); >> return 0; >> } >> compiled with mpicc >> >> Since I know it has been an issue in the past the shell I am using is >> GNU bash, version 3.2.51(1)-release (x86_64-apple-darwin13) >> Copyright (C) 2007 Free Software Foundation, Inc. >> >> any help you can offer would be much appreciated. >> >> Thank you, >> -Drew Lewis >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0