January 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Can't receive messages
by Jeff Hammond 01 Jan '14

01 Jan '14

On Wed, Jan 1, 2014 at 6:19 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On Jan 1, 2014, at 5:58 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> What is this ring you keep referring to? There’s no ring in hydra. Setting one up will lead to more trouble than be useful, with inter-proxy connections and such. I bet that’ll create more queries on this mailing list than help. >> >> A ring is the minimum spanning topology required to verify >> homogeneity. Feel free to use something less efficient if that's >> already available in Hydra. I assume that Hydra proves each process >> the ability to communicate with _at least_ one process outside its >> node and that the connections it has the ability to establish have the >> ability to connect all nodes, otherwise MPI_Init wouldn't be possible. > > Hydra does not do any of those connections. That’s done by the mpich library. > > Furthermore, the hydra proxies starting and communicating with mpiexec already assumes homogeneity. This might not work for heterogeneous platforms at all. So I can’t setup a ring or any other topology. > >> My fundamental conviction is that Hydra could verify homogeneity by >> exchanging and comparing a magic value created by the hash on the set >> independent datatype sizes at negligible overhead compared to whatever >> bootstrap it does already. > > No, it can’t. How will hydra exchange any information efficiently without starting a proxy? The proxies and mpiexec cannot communicate with each other in a heterogeneous environment. The only thing that is heterogeneous safe is “ssh” (or some other launcher). The moment I use ssh to spawn a proxy and expect the proxy to talk to either mpiexec or another proxy, I’m already assuming heterogeneity. > > The only thing you can do is have mpiexec ssh to each node, and get the information needed. This is (1) not scalable, and (2) not viable unless we start distributing a program that gets the datatype information for all datatypes we care about and compare the bits inside mpiexec. Can't you do this in a tree rather than root-to-all? Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] Can't receive messages
by Pavan Balaji 01 Jan '14

01 Jan '14

On Jan 1, 2014, at 5:58 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> What is this ring you keep referring to? There’s no ring in hydra. Setting one up will lead to more trouble than be useful, with inter-proxy connections and such. I bet that’ll create more queries on this mailing list than help. > > A ring is the minimum spanning topology required to verify > homogeneity. Feel free to use something less efficient if that's > already available in Hydra. I assume that Hydra proves each process > the ability to communicate with _at least_ one process outside its > node and that the connections it has the ability to establish have the > ability to connect all nodes, otherwise MPI_Init wouldn't be possible. Hydra does not do any of those connections. That’s done by the mpich library. Furthermore, the hydra proxies starting and communicating with mpiexec already assumes homogeneity. This might not work for heterogeneous platforms at all. So I can’t setup a ring or any other topology. > My fundamental conviction is that Hydra could verify homogeneity by > exchanging and comparing a magic value created by the hash on the set > independent datatype sizes at negligible overhead compared to whatever > bootstrap it does already. No, it can’t. How will hydra exchange any information efficiently without starting a proxy? The proxies and mpiexec cannot communicate with each other in a heterogeneous environment. The only thing that is heterogeneous safe is “ssh” (or some other launcher). The moment I use ssh to spawn a proxy and expect the proxy to talk to either mpiexec or another proxy, I’m already assuming heterogeneity. The only thing you can do is have mpiexec ssh to each node, and get the information needed. This is (1) not scalable, and (2) not viable unless we start distributing a program that gets the datatype information for all datatypes we care about and compare the bits inside mpiexec. — Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Can't receive messages
by Jeff Hammond 01 Jan '14

01 Jan '14

On Wed, Jan 1, 2014 at 4:34 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On Jan 1, 2014, at 4:09 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> 3. Doing an extra ssh to all the nodes is expensive to do every time. >> >> You just need a ring to verify homogeneity. Forming a ring is O(1) per node. > > What is this ring you keep referring to? There’s no ring in hydra. Setting one up will lead to more trouble than be useful, with inter-proxy connections and such. I bet that’ll create more queries on this mailing list than help. A ring is the minimum spanning topology required to verify homogeneity. Feel free to use something less efficient if that's already available in Hydra. I assume that Hydra proves each process the ability to communicate with _at least_ one process outside its node and that the connections it has the ability to establish have the ability to connect all nodes, otherwise MPI_Init wouldn't be possible. My fundamental conviction is that Hydra could verify homogeneity by exchanging and comparing a magic value created by the hash on the set independent datatype sizes at negligible overhead compared to whatever bootstrap it does already. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] Can't receive messages
by Pavan Balaji 01 Jan '14

01 Jan '14

On Jan 1, 2014, at 4:09 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> 3. Doing an extra ssh to all the nodes is expensive to do every time. > > You just need a ring to verify homogeneity. Forming a ring is O(1) per node. What is this ring you keep referring to? There’s no ring in hydra. Setting one up will lead to more trouble than be useful, with inter-proxy connections and such. I bet that’ll create more queries on this mailing list than help. — Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Can't receive messages
by Jeff Hammond 01 Jan '14

01 Jan '14

Sent from my iPhone > On Jan 1, 2014, at 3:17 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > > Most often, this is not a problem on “well setup” clusters that have a resource manager installed on them. While not exclusive, 99% of the time, we see this problem when someone is trying to run mpich between their laptop and some remote server. > It comes up often enough on the list to justify detecting the broken case. Jeff > — Pavan > >> On Jan 1, 2014, at 2:00 PM, Reuti <reuti(a)staff.uni-marburg.de> wrote: >> >>> Am 01.01.2014 um 20:25 schrieb Jeff Hammond: >>> >>> Isn't the process manager infrastructure hetero-safe? At the very least, ssh-ing "uname -a" around the ring of procs can identify the problem on O(1) cost. >> >> As there are clusters without `ssh` availability (i.e. MPICH is running tightly integrated in a queuing system), I would put this in the initial startup call to the slave daemons: `hydra_pmi_proxy` could get an additional parameter about the architecture of the machine where `mpiexec` was issued. It could then return instantly with an error code for something like "refuse to participate". >> >> -- Reuti >> >> >>> Jeff >>> >>> Sent from my iPhone >>> >>>> On Jan 1, 2014, at 11:33 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>> >>>> >>>> If we detect what the local architecture is, how will we communicate it to other processes without having some infrastructure that is hetergeneous-architecture safe? >>>> >>>> — Pavan >>>> >>>>> On Jan 1, 2014, at 9:50 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>> >>>>> One wonders if MPICH could verify the system was sufficiently homogeneous to function properly during initialization rather than defer detection to an error during communication calls. >>>>> >>>>> Jeff >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Jan 1, 2014, at 8:07 AM, Matthias Neuer <mneuer(a)web.de> wrote: >>>>>> >>>>>> Am 31.12.2013 18:44, schrieb Pavan Balaji: >>>>>>> >>>>>>> On Dec 31, 2013, at 10:11 AM, Matthias Neuer <mneuer(a)web.de> wrote: >>>>>>> >>>>>>>>> On 12/31/2013 04:34 PM, Pavan Balaji wrote: >>>>>>>>> >>>>>>>>> Are matze-debian and notebook of the same architecture (x86_64 and running the same OS/configuration)? >>>>>>>> >>>>>>>> No, notebook is x86_32 on debian stable and matze-debian is x86_64 on debian testing, but I compiled the program for each system separately. >>>>>>> >>>>>>> Such heterogeneous configurations are not supported in mpich. >>>>>> >>>>>> Ok, thats the problem then. I didn't find that information in the FAQ or >>>>>> in the installation manual. >>>>>> >>>>>> Thanks for your help. >>>>>> Matthias >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> -- >>>> Pavan Balaji >>>> http://www.mcs.anl.gov/~balaji >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Can't receive messages
by Jeff Hammond 01 Jan '14

01 Jan '14

Sent from my iPhone > On Jan 1, 2014, at 1:50 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > >> On Jan 1, 2014, at 1:25 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> Isn't the process manager infrastructure hetero-safe? At the very least, ssh-ing "uname -a" around the ring of procs can identify the problem on O(1) cost. > > 1. The process manager not heterogeneous-safe. Is it possible to get the proc table sufficient for 3 though? > 2. "uname -a" is not an accurate representation, since two different Linux distributions on the same architecture are not considered heterogeneous. The datatype representation has to be different. > Well query whatever you need to query then. MPICH could hash all the datatypes sizes during configuration and store that value somewhere useful. > 3. Doing an extra ssh to all the nodes is expensive to do every time. > You just need a ring to verify homogeneity. Forming a ring is O(1) per node. Jeff > — Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Can't receive messages
by Pavan Balaji 01 Jan '14

01 Jan '14

Most often, this is not a problem on “well setup” clusters that have a resource manager installed on them. While not exclusive, 99% of the time, we see this problem when someone is trying to run mpich between their laptop and some remote server. — Pavan On Jan 1, 2014, at 2:00 PM, Reuti <reuti(a)staff.uni-marburg.de> wrote: > Am 01.01.2014 um 20:25 schrieb Jeff Hammond: > >> Isn't the process manager infrastructure hetero-safe? At the very least, ssh-ing "uname -a" around the ring of procs can identify the problem on O(1) cost. > > As there are clusters without `ssh` availability (i.e. MPICH is running tightly integrated in a queuing system), I would put this in the initial startup call to the slave daemons: `hydra_pmi_proxy` could get an additional parameter about the architecture of the machine where `mpiexec` was issued. It could then return instantly with an error code for something like "refuse to participate". > > -- Reuti > > >> Jeff >> >> Sent from my iPhone >> >>> On Jan 1, 2014, at 11:33 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>> >>> >>> If we detect what the local architecture is, how will we communicate it to other processes without having some infrastructure that is hetergeneous-architecture safe? >>> >>> — Pavan >>> >>>> On Jan 1, 2014, at 9:50 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>> >>>> One wonders if MPICH could verify the system was sufficiently homogeneous to function properly during initialization rather than defer detection to an error during communication calls. >>>> >>>> Jeff >>>> >>>> Sent from my iPhone >>>> >>>>> On Jan 1, 2014, at 8:07 AM, Matthias Neuer <mneuer(a)web.de> wrote: >>>>> >>>>> Am 31.12.2013 18:44, schrieb Pavan Balaji: >>>>>> >>>>>> On Dec 31, 2013, at 10:11 AM, Matthias Neuer <mneuer(a)web.de> wrote: >>>>>> >>>>>>>> On 12/31/2013 04:34 PM, Pavan Balaji wrote: >>>>>>>> >>>>>>>> Are matze-debian and notebook of the same architecture (x86_64 and running the same OS/configuration)? >>>>>>> >>>>>>> No, notebook is x86_32 on debian stable and matze-debian is x86_64 on debian testing, but I compiled the program for each system separately. >>>>>> >>>>>> Such heterogeneous configurations are not supported in mpich. >>>>> >>>>> Ok, thats the problem then. I didn't find that information in the FAQ or >>>>> in the installation manual. >>>>> >>>>> Thanks for your help. >>>>> Matthias >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Potentially odd memory behavior in MPI_Waitall
by John Grime 01 Jan '14

01 Jan '14

Hi all, I was wondering if someone could help me out with trying to understand what’s gong on with an MPI_Waitall() call with regards to resident set memory usage. Platform information at the end of the email. If I run a 4 process MPI job on my desktop machine, I get a steady increase in memory usage over the course of the program. Apart from the places where I would reasonably expect expect memory usage to increase, I also see regular increases when I call MPI_Waitall(). I measure this behavior via: getmem( &rss0, &vs ); result = MPI_Waitall( n_buffer_reqs, buffer_reqs, MPI_STATUSES_IGNORE ); if( result != MPI_SUCCESS ) ERROR_MACRO( "MPI_Waitall" ); getmem( &rss1, &vs ); … where getmem() queries the OS kernel to return the resident set size (“rss") and virtual memory size (“vs", ignored). The getmem() function does not allocate any memory itself, so I can hopefully see any difference in the resident set memory used before and after the MPI_Waitall() call via (rss1-rss0). After MPI_Waitall() returns, I very often see an increase in the resident set size ranging from 4096 bytes to 65536 bytes. Running an “identical” 4-process job on a single node of the Blue Waters compute cluster (Cray XE system, GCC 4.8.1, Cray mpich 6.0.1) does not seem to produce this behaviour. Switching the MPI_Waitall() for a loop over MPI_Waitany() calls also produces resident set increases. Does anyone have any comments or suggestions as to how I might understand whether this behavior is errant, and how I might track down the cause? It’s irritating me as I’d really like to produce code with as low a memory overhead as possible! Cheers, J. ------------------ Platform info: ------------------ OS: Apple OSX 10.9.1 mpicxx —version: Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) Target: x86_64-apple-darwin13.0.0 Thread model: posix mpiexec —version HYDRA build details: Version: 3.0.4 Release Date: Wed Apr 24 10:08:10 CDT 2013 CC: /usr/bin/clang -pipe -arch x86_64 -Wl,-headerpad_max_install_names -arch x86_64 CXX: /usr/bin/clang++ -pipe -arch x86_64 -Wl,-headerpad_max_install_names -arch x86_64 F77: /opt/local/bin/gfortran-mp-4.8 -pipe -m64 -Wl,-headerpad_max_install_names -arch x86_64 F90: /opt/local/bin/gfortran-mp-4.8 -pipe -m64 -Wl,-headerpad_max_install_names -arch x86_64 Configure options: '--disable-option-checking' '--prefix=/opt/local' '--disable-dependency-tracking' '--disable-silent-rules' '--enable-base-cache' '--enable-cache' '--enable-cxx' '--enable-fast=O2' '--enable-shared' '--enable-smpcoll' '--with-device=ch3:nemesis' '--with-pm=hydra' '--with-thread-package=posix' '--enable-versioning' 'F90FLAGS=' 'F90=' '--enable-timer-type=mach_absolute_time' '--libdir=/opt/local/lib/mpich-mp' '--sysconfdir=/opt/local/etc/mpich-mp' '--program-suffix=-mp' '--enable-f77' '--enable-fc' 'CC=/usr/bin/clang' 'CFLAGS=-pipe -arch x86_64 -O2' 'LDFLAGS=-Wl,-headerpad_max_install_names -arch x86_64 ' 'CXX=/usr/bin/clang++' 'CXXFLAGS=-pipe -arch x86_64 -O2' 'F77=/opt/local/bin/gfortran-mp-4.8' 'FFLAGS=-pipe -m64 -O2' 'FC=/opt/local/bin/gfortran-mp-4.8' 'FCFLAGS=-pipe -m64 -O2' '--cache-file=/dev/null' '--srcdir=.' 'LIBS=-lpthread ' 'CPPFLAGS= -I/opt/local/var/macports/build/_opt_mports_dports_science_mpich/mpich-default/work/mpich-3.0.4/src/mpl/include -I/opt/local/var/macports/build/_opt_mports_dports_science_mpich/mpich-default/work/mpich-3.0.4/src/mpl/include -I/opt/local/var/macports/build/_opt_mports_dports_science_mpich/mpich-default/work/mpich-3.0.4/src/openpa/src -I/opt/local/var/macports/build/_opt_mports_dports_science_mpich/mpich-default/work/mpich-3.0.4/src/openpa/src -I/opt/local/var/macports/build/_opt_mports_dports_science_mpich/mpich-default/work/mpich-3.0.4/src/mpi/romio/include' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Checkpointing libraries available: Demux engines available: poll select

1 0

Re: [mpich-discuss] Can't receive messages
by Reuti 01 Jan '14

01 Jan '14

Am 01.01.2014 um 20:25 schrieb Jeff Hammond: > Isn't the process manager infrastructure hetero-safe? At the very least, ssh-ing "uname -a" around the ring of procs can identify the problem on O(1) cost. As there are clusters without `ssh` availability (i.e. MPICH is running tightly integrated in a queuing system), I would put this in the initial startup call to the slave daemons: `hydra_pmi_proxy` could get an additional parameter about the architecture of the machine where `mpiexec` was issued. It could then return instantly with an error code for something like "refuse to participate". -- Reuti > Jeff > > Sent from my iPhone > >> On Jan 1, 2014, at 11:33 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >> >> If we detect what the local architecture is, how will we communicate it to other processes without having some infrastructure that is hetergeneous-architecture safe? >> >> — Pavan >> >>> On Jan 1, 2014, at 9:50 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> >>> One wonders if MPICH could verify the system was sufficiently homogeneous to function properly during initialization rather than defer detection to an error during communication calls. >>> >>> Jeff >>> >>> Sent from my iPhone >>> >>>> On Jan 1, 2014, at 8:07 AM, Matthias Neuer <mneuer(a)web.de> wrote: >>>> >>>> Am 31.12.2013 18:44, schrieb Pavan Balaji: >>>>> >>>>> On Dec 31, 2013, at 10:11 AM, Matthias Neuer <mneuer(a)web.de> wrote: >>>>> >>>>>>> On 12/31/2013 04:34 PM, Pavan Balaji wrote: >>>>>>> >>>>>>> Are matze-debian and notebook of the same architecture (x86_64 and running the same OS/configuration)? >>>>>> >>>>>> No, notebook is x86_32 on debian stable and matze-debian is x86_64 on debian testing, but I compiled the program for each system separately. >>>>> >>>>> Such heterogeneous configurations are not supported in mpich. >>>> >>>> Ok, thats the problem then. I didn't find that information in the FAQ or >>>> in the installation manual. >>>> >>>> Thanks for your help. >>>> Matthias >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Can't receive messages
by Pavan Balaji 01 Jan '14

01 Jan '14

On Jan 1, 2014, at 1:25 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Isn't the process manager infrastructure hetero-safe? At the very least, ssh-ing "uname -a" around the ring of procs can identify the problem on O(1) cost. 1. The process manager not heterogeneous-safe. 2. "uname -a" is not an accurate representation, since two different Linux distributions on the same architecture are not considered heterogeneous. The datatype representation has to be different. 3. Doing an extra ssh to all the nodes is expensive to do every time. — Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0