June 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] mpirun not running in batch mode.
by Reuti 11 Jun '14

11 Jun '14

Hi, Am 11.06.2014 um 10:55 schrieb Francisco Pastor: > Hi mpich users > > I know I am using an old version of mpich but this one has been working for a long time so when building a new cluster we decided to use the same one. The point is that we have installed RAMS meteorological model with mpich2-1.0.5 and it runs fine when running from console but not with cron or at. Is it running at root then or by a normal user? The path may be different when starting with these methodes too. As it's a new cluster: did you look into the option to install a queuing system? -- Reuti > If the script that runs the model is scheduled with cron it just starts processes on the main node but not in the rest of the cluster nodes. > > While waiting for your help I will try to upgrade to a newer mpich version. > > I can't find the reason why this is happening, probably I made some mistake. Can you help me? Any idea will be welcome and tried. > > Following I attach some information that could be useful to understand my problem. > > meteo@ventus:~$ mpich2version > Version: 1.0.5 > Device: ch3:sock > Configure Options: > CC: gcc > CXX: c++ > F77: f95 > F90: f95 > > > Processes in the main node when run with `at` > > meteo@ventus:~/RAMS/RUN$ ps -ef | grep rams > meteo 20324 18460 0 09:32 ? 00:00:00 python2 /usr/local/mpich2-1.0.5p4/bin/mpirun -envall -n 40 ./rams60 -f RAMSIN.operatiu > meteo 20331 20326 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu > meteo 20332 20327 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu > meteo 20333 20328 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu > meteo 20334 20329 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu > meteo 20335 20330 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu > meteo 20336 20325 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu > meteo 20343 18113 0 09:33 pts/4 00:00:00 grep --color=auto rams > > log of mpirun > > meteo@ventus:~/RAMS/RUN$ cat log_rams_operatiu-INITIAL-20140611-0000 > in main 1024 > numarg: 2 > in main 1024 > numarg: 2 > in main 1024 > numarg: 2 > in main 1024 > numarg: 2 > in main 1024 > numarg: 2 > in main 1024 > numarg: 2 > in main 1024 > numarg: 2 > in main 1024 > numarg: 2 > par init numargs: 3 ./rams60 1024 0 > par init args: 0 argvp[i] ./rams60 > par init args: 1 argvp[i] -f > par init args: 2 argvp[i] RAMSIN.operatiu > par init RAMS_MPI defined > par init numargs: 3 ./rams60 1024 0 > par init args: 0 argvp[i] ./rams60 > par init args: 1 argvp[i] -f > par init args: 2 argvp[i] RAMSIN.operatiu > par init RAMS_MPI defined > par init numargs: 3 ./rams60 1024 0 > par init args: 0 argvp[i] ./rams60 > par init args: 1 argvp[i] -f > par init args: 2 argvp[i] RAMSIN.operatiu > par init RAMS_MPI defined > par init numargs: 3 ./rams60 1024 0 > par init args: 0 argvp[i] ./rams60 > par init args: 1 argvp[i] -f > par init args: 2 argvp[i] RAMSIN.operatiu > par init RAMS_MPI defined > par init numargs: 3 ./rams60 1024 0 > par init args: 0 argvp[i] ./rams60 > par init args: 1 argvp[i] -f > par init args: 2 argvp[i] RAMSIN.operatiu > par init RAMS_MPI defined > par init numargs: 3 ./rams60 1024 0 > par init args: 0 argvp[i] ./rams60 > par init args: 1 argvp[i] -f > par init args: 2 argvp[i] RAMSIN.operatiu > par init RAMS_MPI defined > par init numargs: 3 ./rams60 1024 0 > par init args: 0 argvp[i] ./rams60 > par init args: 1 argvp[i] -f > par init args: 2 argvp[i] RAMSIN.operatiu > par init RAMS_MPI defined > par init numargs: 3 ./rams60 1024 0 > par init args: 0 argvp[i] ./rams60 > par init args: 1 argvp[i] -f > par init args: 2 argvp[i] RAMSIN.operatiu > par init RAMS_MPI defined > > Mpich options to compile RAMS model > > MPI_PATH=/usr/local/mpich2-1.0.5p4 > PAR_INCS=-I$(MPI_PATH)/src/include > PAR_LIBS=-L$(MPI_PATH)/lib/ -lmpich > PAR_DEFS=-DRAMS_MPI > > Thank you very much for your help. > > -- > ----------- > Dr. Francisco Pastor > Meteorology department, Instituto Universitario CEAM-UMH > http://www.ceam.es > ----------- > Mendeley profile: http://www.mendeley.com/profiles/francisco-pastor1/ > Google Scholar: http://scholar.google.com/citations?user=V3mmCdkAAAAJ&hl=es > Researcher ID: http://www.researcherid.com/rid/B-8331-2008 > Cosis profile: http://www.cosis.net/profile/francisco.pastor > ----------- > mail: paco(a)ceam.es > skype: paco.pastor.guzman > ----------- > Parque Tecnologico, C/ Charles R. Darwin, 14 > 46980 PATERNA (Valencia), Spain > Tlf. 96 131 82 27 - Fax. 96 131 81 90 > > > --------------------------------------------------------------------- > Este mensaje y los ficheros anexos son confidenciales. Los mismos contienen información reservada de la empresa que no puede ser difundida. Si usted ha recibido este correo por error, tenga la amabilidad de eliminarlo de su sistema y avisar al remitente mediante reenvío a su dirección electrónica; no deberá copiar el mensaje ni divulgar su contenido a ninguna persona. > > Su dirección de correo electrónico junto a sus datos personales forman parte de un fichero titularidad de la Fundación de la Comunidad Valenciana Centro de Estudios Ambientales del Mediterráneo - CEAM, con CIF: G-46957213, cuya finalidad es la de mantener el contacto con Ud. De acuerdo con la Ley Orgánica 15/1999, usted puede ejercitar sus derechos de acceso, rectificación, cancelación y, en su caso, oposición enviando una solicitud por escrito, acompañada de una fotocopia de su DNI dirigida a: Fundación de la Comunidad Valenciana Centro de Estudios Ambientales del Mediterráneo - CEAM. C/ Charles R. Darwin, 14. Parque Tecnológico.46980 PATERNA (Valencia). > > This message and the attached files are confidential. They contain reserved information belonging to our centre and are not to be broadcast. If you have received this email by mistake, please delete it from your system and alert the sender by returning it to his/her email address. You must not copy or divulge the contents of the message to anyone. > > Your email address and personal data are included in a file belonging to the Fundación de la Comunidad Valenciana Centro de Estudios Ambientales del Mediterráneo - CEAM, con CIF: G-46957213. The purpose of this file is to allow us to keep in contact with you. In accordance with Organic Law 15/1999, you are permitted to access, rectify, cancel or oppose the contents of this file by submitting a written request, accompanied by a photocopy of your DNI, to: Fundación de la Comunidad Valenciana Centro de Estudios Ambientales del Mediterráneo - CEAM. C/ Charles R. Darwin, 14. Parque Tecnológico.46980 PATERNA (Valencia). > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

mpirun not running in batch mode.
by Francisco Pastor 11 Jun '14

11 Jun '14

Hi mpich users I know I am using an old version of mpich but this one has been working for a long time so when building a new cluster we decided to use the same one. The point is that we have installed RAMS meteorological model with mpich2-1.0.5 and it runs fine when running from console but not with cron or at. If the script that runs the model is scheduled with cron it just starts processes on the main node but not in the rest of the cluster nodes. While waiting for your help I will try to upgrade to a newer mpich version. I can't find the reason why this is happening, probably I made some mistake. Can you help me? Any idea will be welcome and tried. Following I attach some information that could be useful to understand my problem. meteo@ventus:~$ mpich2version Version: 1.0.5 Device: ch3:sock Configure Options: CC: gcc CXX: c++ F77: f95 F90: f95 Processes in the main node when run with `at` meteo@ventus:~/RAMS/RUN$ ps -ef | grep rams meteo 20324 18460 0 09:32 ? 00:00:00 python2 /usr/local/mpich2-1.0.5p4/bin/mpirun -envall -n 40 ./rams60 -f RAMSIN.operatiu meteo 20331 20326 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu meteo 20332 20327 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu meteo 20333 20328 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu meteo 20334 20329 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu meteo 20335 20330 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu meteo 20336 20325 0 09:32 ? 00:00:00 ./rams60 -f RAMSIN.operatiu meteo 20343 18113 0 09:33 pts/4 00:00:00 grep --color=auto rams log of mpirun meteo@ventus:~/RAMS/RUN$ cat log_rams_operatiu-INITIAL-20140611-0000 in main 1024 numarg: 2 in main 1024 numarg: 2 in main 1024 numarg: 2 in main 1024 numarg: 2 in main 1024 numarg: 2 in main 1024 numarg: 2 in main 1024 numarg: 2 in main 1024 numarg: 2 par init numargs: 3 ./rams60 1024 0 par init args: 0 argvp[i] ./rams60 par init args: 1 argvp[i] -f par init args: 2 argvp[i] RAMSIN.operatiu par init RAMS_MPI defined par init numargs: 3 ./rams60 1024 0 par init args: 0 argvp[i] ./rams60 par init args: 1 argvp[i] -f par init args: 2 argvp[i] RAMSIN.operatiu par init RAMS_MPI defined par init numargs: 3 ./rams60 1024 0 par init args: 0 argvp[i] ./rams60 par init args: 1 argvp[i] -f par init args: 2 argvp[i] RAMSIN.operatiu par init RAMS_MPI defined par init numargs: 3 ./rams60 1024 0 par init args: 0 argvp[i] ./rams60 par init args: 1 argvp[i] -f par init args: 2 argvp[i] RAMSIN.operatiu par init RAMS_MPI defined par init numargs: 3 ./rams60 1024 0 par init args: 0 argvp[i] ./rams60 par init args: 1 argvp[i] -f par init args: 2 argvp[i] RAMSIN.operatiu par init RAMS_MPI defined par init numargs: 3 ./rams60 1024 0 par init args: 0 argvp[i] ./rams60 par init args: 1 argvp[i] -f par init args: 2 argvp[i] RAMSIN.operatiu par init RAMS_MPI defined par init numargs: 3 ./rams60 1024 0 par init args: 0 argvp[i] ./rams60 par init args: 1 argvp[i] -f par init args: 2 argvp[i] RAMSIN.operatiu par init RAMS_MPI defined par init numargs: 3 ./rams60 1024 0 par init args: 0 argvp[i] ./rams60 par init args: 1 argvp[i] -f par init args: 2 argvp[i] RAMSIN.operatiu par init RAMS_MPI defined Mpich options to compile RAMS model MPI_PATH=/usr/local/mpich2-1.0.5p4 PAR_INCS=-I$(MPI_PATH)/src/include PAR_LIBS=-L$(MPI_PATH)/lib/ -lmpich PAR_DEFS=-DRAMS_MPI Thank you very much for your help. -- ----------- Dr. Francisco Pastor Meteorology department, Instituto Universitario CEAM-UMH http://www.ceam.es ----------- Mendeley profile: http://www.mendeley.com/profiles/francisco-pastor1/ Google Scholar: http://scholar.google.com/citations?user=V3mmCdkAAAAJ&hl=es Researcher ID: http://www.researcherid.com/rid/B-8331-2008 Cosis profile: http://www.cosis.net/profile/francisco.pastor ----------- mail: paco(a)ceam.es skype: paco.pastor.guzman ----------- Parque Tecnologico, C/ Charles R. Darwin, 14 46980 PATERNA (Valencia), Spain Tlf. 96 131 82 27 - Fax. 96 131 81 90 --------------------------------------------------------------------- Este mensaje y los ficheros anexos son confidenciales. Los mismos contienen informaciï¿½n reservada de la empresa que no puede ser difundida. Si usted ha recibido este correo por error, tenga la amabilidad de eliminarlo de su sistema y avisar al remitente mediante reenvï¿½o a su direcciï¿½n electrï¿½nica; no deberï¿½ copiar el mensaje ni divulgar su contenido a ninguna persona. Su direcciï¿½n de correo electrï¿½nico junto a sus datos personales forman parte de un fichero titularidad de la Fundaciï¿½n de la Comunidad Valenciana Centro de Estudios Ambientales del Mediterrï¿½neo - CEAM, con CIF: G-46957213, cuya finalidad es la de mantener el contacto con Ud. De acuerdo con la Ley Orgï¿½nica 15/1999, usted puede ejercitar sus derechos de acceso, rectificaciï¿½n, cancelaciï¿½n y, en su caso, oposiciï¿½n enviando una solicitud por escrito, acompaï¿½ada de una fotocopia de su DNI dirigida a: Fundaciï¿½n de la Comunidad Valenciana Centro de Estudios Ambientales del Mediterrï¿½neo - CEAM. C/ Charles R. Darwin, 14. Parque Tecnolï¿½gico.46980 PATERNA (Valencia). This message and the attached files are confidential. They contain reserved information belonging to our centre and are not to be broadcast. If you have received this email by mistake, please delete it from your system and alert the sender by returning it to his/her email address. You must not copy or divulge the contents of the message to anyone. Your email address and personal data are included in a file belonging to the Fundaciï¿½n de la Comunidad Valenciana Centro de Estudios Ambientales del Mediterrï¿½neo - CEAM, con CIF: G-46957213. The purpose of this file is to allow us to keep in contact with you. In accordance with Organic Law 15/1999, you are permitted to access, rectify, cancel or oppose the contents of this file by submitting a written request, accompanied by a photocopy of your DNI, to: Fundaciï¿½n de la Comunidad Valenciana Centro de Estudios Ambientales del Mediterrï¿½neo - CEAM. C/ Charles R. Darwin, 14. Parque Tecnolï¿½gico.46980 PATERNA (Valencia).

1 0

Re: [mpich-discuss] Cluster with uneven CPU speeds
by Balaji, Pavan 06 Jun '14

06 Jun '14

You need to declare your global variables as thread private and make other similar changes. It’s not transparent. — Pavan On Jun 6, 2014, at 10:23 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > You may consider AMPI (http://charm.cs.uiuc.edu/ppl_research/ampi/) which supports automatic dynamic load balancing, a very cool idea. > > --Junchao Zhang > > > On Thu, Jun 5, 2014 at 8:51 PM, Ron Palmer <ron.palmer(a)pgcgroup.com.au> wrote: > Pavan, > thanks for your reply and comments. Unfortunately, the actual application I am running is outside my sphere of influence (though as I am running a beta code, I will forward your thoughts in hope they will consider them). > > Is there another way that does not include looking at the actual application doing the work? > > Regards, > Ron > > > On 6/06/2014 11:48, Balaji, Pavan wrote: >> Ron, >> >> In general, oversubscribing the cores of a node is a bad idea. MPI is optimized for the common case where each MPI process is on at least one core, which most applications use. This, however, adds a cost when you oversubscribe, and is not recommended. >> >> To deal with cores that operate at different speeds, the only good way is to restructure your algorithm to be more asynchronous in nature. For example, if a master-worker model is possible, that might work great. Some workers (which are running on faster cores) do more work than others. However, not all algorithms can be expressed in this model. There are other asynchronous models possible too. >> >> In short, I think it’s time to go back to the whiteboard and see if the algorithm used by the application is appropriate or not. >> >> — Pavan >> >> On Jun 5, 2014, at 7:42 PM, Ron Palmer >> <ron.palmer(a)pgcgroup.com.au> >> wrote: >> >> >>> I have a small cluster of computers with uneven clock speed CPUs and currently I am running with "-np" == total number of cores. However, it appears as if the fastest computer has to wait for the slower ones to finish at the end (at least I believe so). The most recent process took 65 hours so I am interested in finding ways to optimise the process. >>> >>> Is it possible to, say, use a larger "-np" and then increase the thread number for the faster CPUs in the machine file to make the faster computers do more work so, ideally, they all finish about the same time? Will it finish off the first batch then start on the next batch? Or, will the faster computers just get more concurrent jobs, possibly slowing down the processing? >>> >>> eg, if the single CPU of PC_A has twice the clock rating to that of single CPU PC_B, and both has quad cores, then use -np=12 and then have the following in the machinefile: >>> PC_A:8 >>> PC_B:4 >>> >>> Perhaps this is something better addressed with job scheduling software like GridEngine? Reuti? >>> >>> Thanks, >>> Ron >>> >>> >>> -- >>> Ron Palmer MSc MBA. >>> Principal Geophysicist >>> >>> ron.palmer(a)pgcgroup.com.au >>> >>> 0413 579 099 >>> 07 3103 4963 >>> >>> >>> _______________________________________________ >>> discuss mailing list >>> discuss(a)mpich.org >>> >>> To manage subscription options or unsubscribe: >>> >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list >> discuss(a)mpich.org >> >> To manage subscription options or unsubscribe: >> >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Ron Palmer MSc MBA. > Principal Geophysicist > ron.palmer(a)pgcgroup.com.au > 0413 579 099 > 07 3103 4963 > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Cluster with uneven CPU speeds
by Junchao Zhang 06 Jun '14

06 Jun '14

You may consider AMPI (http://charm.cs.uiuc.edu/ppl_research/ampi/) which supports *automatic* dynamic load balancing, a very cool idea. --Junchao Zhang On Thu, Jun 5, 2014 at 8:51 PM, Ron Palmer <ron.palmer(a)pgcgroup.com.au> wrote: > Pavan, > thanks for your reply and comments. Unfortunately, the actual application > I am running is outside my sphere of influence (though as I am running a > beta code, I will forward your thoughts in hope they will consider them). > > Is there another way that does not include looking at the actual > application doing the work? > > Regards, > Ron > > > On 6/06/2014 11:48, Balaji, Pavan wrote: > > Ron, > > In general, oversubscribing the cores of a node is a bad idea. MPI is optimized for the common case where each MPI process is on at least one core, which most applications use. This, however, adds a cost when you oversubscribe, and is not recommended. > > To deal with cores that operate at different speeds, the only good way is to restructure your algorithm to be more asynchronous in nature. For example, if a master-worker model is possible, that might work great. Some workers (which are running on faster cores) do more work than others. However, not all algorithms can be expressed in this model. There are other asynchronous models possible too. > > In short, I think it’s time to go back to the whiteboard and see if the algorithm used by the application is appropriate or not. > > — Pavan > > On Jun 5, 2014, at 7:42 PM, Ron Palmer <ron.palmer(a)pgcgroup.com.au> <ron.palmer(a)pgcgroup.com.au> wrote: > > > I have a small cluster of computers with uneven clock speed CPUs and currently I am running with "-np" == total number of cores. However, it appears as if the fastest computer has to wait for the slower ones to finish at the end (at least I believe so). The most recent process took 65 hours so I am interested in finding ways to optimise the process. > > Is it possible to, say, use a larger "-np" and then increase the thread number for the faster CPUs in the machine file to make the faster computers do more work so, ideally, they all finish about the same time? Will it finish off the first batch then start on the next batch? Or, will the faster computers just get more concurrent jobs, possibly slowing down the processing? > > eg, if the single CPU of PC_A has twice the clock rating to that of single CPU PC_B, and both has quad cores, then use -np=12 and then have the following in the machinefile: > PC_A:8 > PC_B:4 > > Perhaps this is something better addressed with job scheduling software like GridEngine? Reuti? > > Thanks, > Ron > > > -- > Ron Palmer MSc MBA. > Principal Geophysicistron.palmer(a)pgcgroup.com.au > 0413 579 099 > 07 3103 4963 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe:https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe:https://lists.mpich.org/mailman/listinfo/discuss > > > -- > > *Ron Palmer* MSc MBA. > > Principal Geophysicist > > ron.palmer(a)pgcgroup.com.au > > 0413 579 099 > > 07 3103 4963 > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Cluster with uneven CPU speeds
by Reuti 06 Jun '14

06 Jun '14

Am 06.06.2014 um 16:03 schrieb Gus Correa: > On 06/06/2014 07:27 AM, Reuti wrote: >> Am 06.06.2014 um 02:42 schrieb Ron Palmer: >> >>> I have a small cluster of computers with uneven clock speed CPUs and currently I am running with "-np" == total number of cores. However, it appears as if the fastest computer has to wait for the slower ones to finish at the end (at least I believe so). The most recent process took 65 hours so I am interested in finding ways to optimise the process. >>> >>> Is it possible to, say, use a larger "-np" and then increase the thread number for the faster CPUs in the machine file to make the faster computers do more work so, ideally, they all finish about the same time? Will it finish off the first batch then start on the next batch? Or, will the faster computers just get more concurrent jobs, possibly slowing down the processing? >>> >>> eg, if the single CPU of PC_A has twice the clock rating to that of single CPU PC_B, and both has quad cores, then use -np=12 and then have the following in the machinefile: >>> PC_A:8 >>> PC_B:4 >>> >>> Perhaps this is something better addressed with job scheduling software like GridEngine? Reuti? >> >> You could define more slots than cores are available in the queue definition for (a) particular node(s), but this would enable the workaround you outlined already in SGE's way only. >> >> The way MPI was designed was having equal computing power per rank in mind I think (and often clusters are designed this way nowadays, even after an upgrade of a cluster buy adding newer/faster nodes you could route the jobs to use only machines of the same kind in SGE). In PVM there was even an entry for the relative speed of a machine in the hostfile (as a collection of different machines was often seen that days), but there is nothing similar in MPI AFAIK. >> >> -- Reuti >> >> >>> Thanks, >>> Ron >>> >>> >>> -- >>> Ron Palmer MSc MBA. >>> Principal Geophysicist >>> ron.palmer(a)pgcgroup.com.au >>> 0413 579 099 >>> 07 3103 4963 > > Hi Ron > > It may be more efficient to just partition the cluster into slow and fast nodes, and direct the jobs to either group (based on queue name, > your group's priorities, etc). > The jobs should then run on groups of similar nodes. > I guess the overall job throughput of these smaller/longer-lived jobs > won't be worse than if you try > to load balance big jobs across processors of different capability (which is not an easy task anyway, as Pavan and Reuti pointed out). > > I am not familiar to SGE, but I guess this partitioning > can be done there. > On Torque what this takes is to declare node's properties > (say, 'slow', 'fast', or '1.8GHz', '2.4GHz') > in the nodes' file and, if desired, > create separate queues that default to nodes with the appropriate In SGE it's better to live with one queue for such a setup and create different PEs like MPI1 and MPI2 which are attached to different hosts (or hostgroups) of the one and only queue. Besides submitting and requesting MPI1 or MPI2 for the PE, one can also request MPI* and SGE will select any of the two (or more) partitions of the cluster - but select nodes from this chosen partition only. -- Reuti > properties, then direct the jobs to the proper queues (or request the > nodes with the specific properties). > > My two cents, > Gus Correa > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Cluster with uneven CPU speeds
by Gus Correa 06 Jun '14

06 Jun '14

On 06/06/2014 07:27 AM, Reuti wrote: > Am 06.06.2014 um 02:42 schrieb Ron Palmer: > >> I have a small cluster of computers with uneven clock speed CPUs and currently I am running with "-np" == total number of cores. However, it appears as if the fastest computer has to wait for the slower ones to finish at the end (at least I believe so). The most recent process took 65 hours so I am interested in finding ways to optimise the process. >> >> Is it possible to, say, use a larger "-np" and then increase the thread number for the faster CPUs in the machine file to make the faster computers do more work so, ideally, they all finish about the same time? Will it finish off the first batch then start on the next batch? Or, will the faster computers just get more concurrent jobs, possibly slowing down the processing? >> >> eg, if the single CPU of PC_A has twice the clock rating to that of single CPU PC_B, and both has quad cores, then use -np=12 and then have the following in the machinefile: >> PC_A:8 >> PC_B:4 >> >> Perhaps this is something better addressed with job scheduling software like GridEngine? Reuti? > > You could define more slots than cores are available in the queue definition for (a) particular node(s), but this would enable the workaround you outlined already in SGE's way only. > > The way MPI was designed was having equal computing power per rank in mind I think (and often clusters are designed this way nowadays, even after an upgrade of a cluster buy adding newer/faster nodes you could route the jobs to use only machines of the same kind in SGE). In PVM there was even an entry for the relative speed of a machine in the hostfile (as a collection of different machines was often seen that days), but there is nothing similar in MPI AFAIK. > > -- Reuti > > >> Thanks, >> Ron >> >> >> -- >> Ron Palmer MSc MBA. >> Principal Geophysicist >> ron.palmer(a)pgcgroup.com.au >> 0413 579 099 >> 07 3103 4963 Hi Ron It may be more efficient to just partition the cluster into slow and fast nodes, and direct the jobs to either group (based on queue name, your group's priorities, etc). The jobs should then run on groups of similar nodes. I guess the overall job throughput of these smaller/longer-lived jobs won't be worse than if you try to load balance big jobs across processors of different capability (which is not an easy task anyway, as Pavan and Reuti pointed out). I am not familiar to SGE, but I guess this partitioning can be done there. On Torque what this takes is to declare node's properties (say, 'slow', 'fast', or '1.8GHz', '2.4GHz') in the nodes' file and, if desired, create separate queues that default to nodes with the appropriate properties, then direct the jobs to the proper queues (or request the nodes with the specific properties). My two cents, Gus Correa

1 0

Re: [mpich-discuss] Cluster with uneven CPU speeds
by Reuti 06 Jun '14

06 Jun '14

Am 06.06.2014 um 02:42 schrieb Ron Palmer: > I have a small cluster of computers with uneven clock speed CPUs and currently I am running with "-np" == total number of cores. However, it appears as if the fastest computer has to wait for the slower ones to finish at the end (at least I believe so). The most recent process took 65 hours so I am interested in finding ways to optimise the process. > > Is it possible to, say, use a larger "-np" and then increase the thread number for the faster CPUs in the machine file to make the faster computers do more work so, ideally, they all finish about the same time? Will it finish off the first batch then start on the next batch? Or, will the faster computers just get more concurrent jobs, possibly slowing down the processing? > > eg, if the single CPU of PC_A has twice the clock rating to that of single CPU PC_B, and both has quad cores, then use -np=12 and then have the following in the machinefile: > PC_A:8 > PC_B:4 > > Perhaps this is something better addressed with job scheduling software like GridEngine? Reuti? You could define more slots than cores are available in the queue definition for (a) particular node(s), but this would enable the workaround you outlined already in SGE's way only. The way MPI was designed was having equal computing power per rank in mind I think (and often clusters are designed this way nowadays, even after an upgrade of a cluster buy adding newer/faster nodes you could route the jobs to use only machines of the same kind in SGE). In PVM there was even an entry for the relative speed of a machine in the hostfile (as a collection of different machines was often seen that days), but there is nothing similar in MPI AFAIK. -- Reuti > Thanks, > Ron > > > -- > Ron Palmer MSc MBA. > Principal Geophysicist > ron.palmer(a)pgcgroup.com.au > 0413 579 099 > 07 3103 4963 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Cluster with uneven CPU speeds
by Jed Brown 06 Jun '14

06 Jun '14

"Balaji, Pavan" <balaji(a)anl.gov> writes: > If you want to try the approach with oversubscription, I’d recommend > configuring mpich using --with-device=ch3:sock. That build will not > be as optimized for the common case of dedicated cores for each MPI > process, but also won't hurt the oversubscription case as much. I > already warned you that this is a bad idea, so use at your own risk. > :-) It's a bad idea for performance, but great for testing and debugging. It's crazy to not be able to run a 9-process correctness test on a laptop. For this reason, PETSc's --download-mpich feature (used by people that don't have MPI installed, or whose OS provides a broken MPI) still uses ch3:sock by default.

1 0

Re: [mpich-discuss] Cluster with uneven CPU speeds
by Balaji, Pavan 05 Jun '14

05 Jun '14

On Jun 5, 2014, at 8:51 PM, Ron Palmer <ron.palmer(a)pgcgroup.com.au> wrote: > Is there another way that does not include looking at the actual application doing the work? No good solution comes to mind. If you want to try the approach with oversubscription, I’d recommend configuring mpich using --with-device=ch3:sock. That build will not be as optimized for the common case of dedicated cores for each MPI process, but also won't hurt the oversubscription case as much. I already warned you that this is a bad idea, so use at your own risk. :-) — Pavan

1 0

Re: [mpich-discuss] Cluster with uneven CPU speeds
by Ron Palmer 05 Jun '14

05 Jun '14

Pavan, thanks for your reply and comments. Unfortunately, the actual application I am running is outside my sphere of influence (though as I am running a beta code, I will forward your thoughts in hope they will consider them). Is there another way that does not include looking at the actual application doing the work? Regards, Ron On 6/06/2014 11:48, Balaji, Pavan wrote: > Ron, > > In general, oversubscribing the cores of a node is a bad idea. MPI is optimized for the common case where each MPI process is on at least one core, which most applications use. This, however, adds a cost when you oversubscribe, and is not recommended. > > To deal with cores that operate at different speeds, the only good way is to restructure your algorithm to be more asynchronous in nature. For example, if a master-worker model is possible, that might work great. Some workers (which are running on faster cores) do more work than others. However, not all algorithms can be expressed in this model. There are other asynchronous models possible too. > > In short, I think itï¿½s time to go back to the whiteboard and see if the algorithm used by the application is appropriate or not. > > ï¿½ Pavan > > On Jun 5, 2014, at 7:42 PM, Ron Palmer <ron.palmer(a)pgcgroup.com.au> wrote: > >> I have a small cluster of computers with uneven clock speed CPUs and currently I am running with "-np" == total number of cores. However, it appears as if the fastest computer has to wait for the slower ones to finish at the end (at least I believe so). The most recent process took 65 hours so I am interested in finding ways to optimise the process. >> >> Is it possible to, say, use a larger "-np" and then increase the thread number for the faster CPUs in the machine file to make the faster computers do more work so, ideally, they all finish about the same time? Will it finish off the first batch then start on the next batch? Or, will the faster computers just get more concurrent jobs, possibly slowing down the processing? >> >> eg, if the single CPU of PC_A has twice the clock rating to that of single CPU PC_B, and both has quad cores, then use -np=12 and then have the following in the machinefile: >> PC_A:8 >> PC_B:4 >> >> Perhaps this is something better addressed with job scheduling software like GridEngine? Reuti? >> >> Thanks, >> Ron >> >> >> -- >> Ron Palmer MSc MBA. >> Principal Geophysicist >> ron.palmer(a)pgcgroup.com.au >> 0413 579 099 >> 07 3103 4963 >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- *Ron Palmer*MSc MBA. Principal Geophysicist ron.palmer(a)pgcgroup.com.au <mailto:[email protected]> 0413 579 099 07 3103 4963

1 0