June 2015 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_Allreduce over multiple machines
by Winkler, Ursula (ursula.winkler＠uni-graz.at) 26 Jun '15

26 Jun '15

Hi Gordon, did you already check the firewall settings? Regards, Ursula Von: Gordon Reid [mailto:[email protected]] Gesendet: Donnerstag, 25. Juni 2015 15:46 An: discuss(a)mpich.org Betreff: [mpich-discuss] MPI_Allreduce over multiple machines Hello, I was wondering if somebody would give me assistance on a question I have regarding MPI_Allreduce. Details are on stackoverflow for brevity: http://stackoverflow.com/questions/30955796/mpi-allreduce-error-on-mpich-3-… Thanks in advance, Gordon _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Allreduce over multiple machines
by Thakur, Rajeev 25 Jun '15

25 Jun '15

Does it also happen at lower iteration counts? Rajeev On Jun 25, 2015, at 8:46 AM, Gordon Reid <gordon.reid1992(a)hotmail.co.uk> wrote: > Hello, > > I was wondering if somebody would give me assistance on a question I have regarding MPI_Allreduce. Details are on stackoverflow for brevity: http://stackoverflow.com/questions/30955796/mpi-allreduce-error-on-mpich-3-… > > Thanks in advance, > Gordon > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_Allreduce over multiple machines
by Gordon Reid 25 Jun '15

25 Jun '15

Hello, I was wondering if somebody would give me assistance on a question I have regarding MPI_Allreduce. Details are on stackoverflow for brevity: http://stackoverflow.com/questions/30955796/mpi-allreduce-error-on-mpich-3-… Thanks in advance,Gordon _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] resource allocation and multiple mpi_comm_spawn's
by Junchao Zhang 16 Jun '15

16 Jun '15

A ticket is created for it, https://trac.mpich.org/projects/mpich/ticket/2277 Thanks for your patch. --Junchao Zhang On Tue, Jun 16, 2015 at 7:43 AM, Arjen van Elteren <info(a)arjenvanelteren.com > wrote: > Hello, > > I've patched the hydra process manager (mpich release 3.1), to support > my case (see attachment). > > It's a bit ugly (I have not prefixed the sort and compare function names > as these are local to the alloc.c file) but it works for me and makes > the resource allocation for consecutive MPI_COMM_SPAWNs more fair. > > All I had to patch was 'utils/alloc/alloc.c' in the hydra source-code. > > New procedure for allocating proxy's and assigning executables is now: > > 1. make proxies (NEW: make a proxy for every node, do not stop when > number of processes is reached) > 2 (NEW). sort the proxies by their nodes active number of processes > (decreasing, the first proxy will have the least number of processes) > 3. allocate executables to the proxies > > Should I create a bug report for this? (I can't find a login button on > the trac website) > > Kind regards, > > Arjen > > On 12-06-15 11:23, Arjen van Elteren wrote: > > Hello, > > > > I'm working with an application that invokes multiple mpi_comm_spawn > calls. > > > > I'm using mpiexec on a cluster without a resource manager or job queue, > > so plain ssh and fork calls and everything is managed by mpich. > > > > It looks like mpiexec (both hydra and mpd) re-use the hostfile from the > > start (and do not look at already allocated resources/used nodes). > > > > For example I have a hostfile like this: > > > > node01:1 > > node02:1 > > node03:1 > > > > When I run a call like this: > > > > MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, > > MPI_INFO_NULL, 2, > > MPI_COMM_SELF, &worker, > > MPI_ERRCODES_IGNORE); > > > > I get an allocation like this: > > > > node process > > --------------- ------------------ > > node01 manager > > node02 worker 1 > > node03 worker 2 > > > > Which is what I expected. > > > > But when I instead do 2 calls like this (i.e. each worker has one > > process, but there are 2 workers): > > > > MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, > > MPI_INFO_NULL, 1, > > MPI_COMM_SELF, &worker, > > MPI_ERRCODES_IGNORE); > > > > MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, > > MPI_INFO_NULL, 1, > > MPI_COMM_SELF, &worker, > > MPI_ERRCODES_IGNORE); > > > > I get an allocation like this (both hydra and mpd): > > > > node process > > --------------- ------------------ > > node01 manager > > node02 worker 1 + worker 2 > > node03 > > > > Which is not what I expected at all! > > > > In fact, when I do this for a more complex example, I conclude that in > > MPI_Comm_spawn the hostfile is simply re-interpreted for every spawn and > > previous allocations in the same application are not accounted for. > > > > I know I could set hostname in the MPI_Comm_spawn call, but then I'm > > moving deployment information into my application (and I don't want to > > recompile or add a commandline argument for something that should be > > handled by mpiexec) > > > > Is there an option or easy fix for this problem? (I looked at the code > > of hydra, but I'm unsure how the different proxy's and processes divide > > this spawning work between them (I could not easily detect one "grand > > master" that does the allocation...) > > > > Kind regards, > > > > Arjen > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] resource allocation and multiple mpi_comm_spawn's
by Arjen van Elteren 16 Jun '15

16 Jun '15

Hello, I've patched the hydra process manager (mpich release 3.1), to support my case (see attachment). It's a bit ugly (I have not prefixed the sort and compare function names as these are local to the alloc.c file) but it works for me and makes the resource allocation for consecutive MPI_COMM_SPAWNs more fair. All I had to patch was 'utils/alloc/alloc.c' in the hydra source-code. New procedure for allocating proxy's and assigning executables is now: 1. make proxies (NEW: make a proxy for every node, do not stop when number of processes is reached) 2 (NEW). sort the proxies by their nodes active number of processes (decreasing, the first proxy will have the least number of processes) 3. allocate executables to the proxies Should I create a bug report for this? (I can't find a login button on the trac website) Kind regards, Arjen On 12-06-15 11:23, Arjen van Elteren wrote: > Hello, > > I'm working with an application that invokes multiple mpi_comm_spawn calls. > > I'm using mpiexec on a cluster without a resource manager or job queue, > so plain ssh and fork calls and everything is managed by mpich. > > It looks like mpiexec (both hydra and mpd) re-use the hostfile from the > start (and do not look at already allocated resources/used nodes). > > For example I have a hostfile like this: > > node01:1 > node02:1 > node03:1 > > When I run a call like this: > > MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, > MPI_INFO_NULL, 2, > MPI_COMM_SELF, &worker, > MPI_ERRCODES_IGNORE); > > I get an allocation like this: > > node process > --------------- ------------------ > node01 manager > node02 worker 1 > node03 worker 2 > > Which is what I expected. > > But when I instead do 2 calls like this (i.e. each worker has one > process, but there are 2 workers): > > MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, > MPI_INFO_NULL, 1, > MPI_COMM_SELF, &worker, > MPI_ERRCODES_IGNORE); > > MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, > MPI_INFO_NULL, 1, > MPI_COMM_SELF, &worker, > MPI_ERRCODES_IGNORE); > > I get an allocation like this (both hydra and mpd): > > node process > --------------- ------------------ > node01 manager > node02 worker 1 + worker 2 > node03 > > Which is not what I expected at all! > > In fact, when I do this for a more complex example, I conclude that in > MPI_Comm_spawn the hostfile is simply re-interpreted for every spawn and > previous allocations in the same application are not accounted for. > > I know I could set hostname in the MPI_Comm_spawn call, but then I'm > moving deployment information into my application (and I don't want to > recompile or add a commandline argument for something that should be > handled by mpiexec) > > Is there an option or easy fix for this problem? (I looked at the code > of hydra, but I'm unsure how the different proxy's and processes divide > this spawning work between them (I could not easily detect one "grand > master" that does the allocation...) > > Kind regards, > > Arjen > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Linux socket command may return 0 until MPI_Init_thread called.
by John DelSignore 15 Jun '15

15 Jun '15

_______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Automake version in MPICH 3.1.4
by Aurélien Larcher 15 Jun '15

15 Jun '15

Hi, Thank you for your answer ! For some reason out of source build of 3.1.4 is broken on my platform, so I need to investigate. Best, Aurelien On Sun, Jun 7, 2015 at 11:10 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > Hi, > > We needed some fixes in automake-1.15, so the upgrade was not arbitrary. > > Having said that, that only introduces a dependency for developers. > Users who download the tarball shouldn't need automake (or any of the > autotools). > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > From: Aurélien Larcher > Reply-To: "discuss(a)mpich.org" > Date: Sunday, June 7, 2015 at 11:46 AM > To: "<discuss(a)mpich.org>" > Subject: [mpich-discuss] Automake version in MPICH 3.1.4 > > Hi, > I just noticed that the following commit on 2015-02-09: > > > http://git.mpich.org/mpich.git/commit/8ac6607db3a087cc8986d8124cab2b0930ae3… > > adds automake >= 1.15 requirement in autogen.sh > > Is this fix really necessary for general platforms or is it platform > specific ? > > Does it not seem a bit too drastic to force a dependency on a core > package released on 2015-01-05 (a month before or so) just to get a fix > which is not mpich dependent ? > > Thank you in avance ! > Best regards > > Aurélien > > -- > > ------------------------------------------------------------------------------- > LARCHER Aurélien | KTH, School of Computer Science and > Communication > Work: +46 (0) 8 790 71 42 | Lindstedtsvägen 5, Plan 4, 100 44 Stockholm, > SWEDEN > > ------------------------------------------------------------------------------- > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- ------------------------------------------------------------------------------- LARCHER Aurélien | KTH, School of Computer Science and Communication Work: +46 (0) 8 790 71 42 | Lindstedtsvägen 5, Plan 4, 100 44 Stockholm, SWEDEN ------------------------------------------------------------------------------- _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Assertion error (enable-thread-cs=per-object)
by "Antonio J. Peña" 15 Jun '15

15 Jun '15

Hi Guillem, Sorry, at this time that option is not supported on the CH3 channel. Best, Antonio On 06/15/2015 06:14 AM, Guillem Blanco wrote: > Hi, > > I am using mpich-3.1.4 configured —enable-thread-cs=per-object. Every time I execute it I get the following assertion error: > > "Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 639: pkt->type >= 0 && pkt->type < MPIDI_CH3_PKT_END_ALL" > > However, if I compile mpich without —enable-thread-cs=per-object everything works correctly. > > Any help on this? > > Thanks, > > Guillem Blanco. > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Application aborts at runtime
by Thakur, Rajeev 15 Jun '15

15 Jun '15

And to query the max tag value, you need to call the function MPI_Comm_get_attr(MPI_COMM_WORLD, MPI_TAG_UB, &value, &flag). Then print “value”. MPI_TAG_UB is a key. Its associated value can be found by calling this function, not by doing a printf on MPI_TAG_UB. Rajeev > On Jun 15, 2015, at 1:45 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > > It's the other way around — the MPI implementation is allowed to pick whatever upper bound it likes. The application is required to query this value before assigning the tag value. Even if MPICH increases this value, some other MPI implementation might not and you'll run into this problem again. > > The bottom line is that your application is not compliant with the MPI standard. You'll need to fix that. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > From: Chaitra Kumar > Reply-To: "discuss(a)mpich.org" > Date: Monday, June 15, 2015 at 12:13 AM > To: "discuss(a)mpich.org" > Subject: Re: [mpich-discuss] Application aborts at runtime > > Hi Rajeev, > > Thanks for the reference to the relevant section of MPI 3.1 standard. > > As mentioned in the standard it specifies only the least value of upper bound which is 32767. An MPI implementation is free to set a high value of MPI_TAG_UB. > > So can this upper bound value (MPI_TAB_UB) be increased to allow higher tag values? > > Regards, > Chaitra > > On Fri, Jun 12, 2015 at 9:31 AM, Thakur, Rajeev <thakur(a)mcs.anl.gov> wrote: > The tag cannot be larger than the value of the attribute key MPI_TAG_UB on MPI_COMM_WORLD. See Section 8.1.2, pg 334, of the MPI 3.1 standard. > > Rajeev > > > On Jun 11, 2015, at 8:17 PM, Chaitra Kumar <chaitragkumar(a)gmail.com> wrote: > > > > Hi Junchao, > > > > Yes, the application has a requirement for big tag. > > > > With openMPI there are no issues while running the application. > > > > Can you please do the needful. > > > > Regards, > > Chaitra > > > > On Thu, Jun 11, 2015 at 5:21 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > > As discussed in the mvapich mailing list, do you really need such big tag? > > > > --Junchao Zhang > > > > On Wed, Jun 10, 2015 at 8:35 PM, Chaitra Kumar <chaitragkumar(a)gmail.com> wrote: > > Hi Team, > > > > I am trying to run an application using MVAPICH2-2.0. The application compiles without any error, but at runtime, aborts with following error: > > > > Fatal error in PMPI_Isend:Invalid tag, error stack:PMPI_Isend(149): MPI_Isend(buf=0x7f59b5124300, count=8, MPI_UNSIGNED_LONG, dest=0, tag=1073741824, comm=0x84000001, request=0x132ae24) failed PMPI_Isend(98).: Invalid tag, value is 1073741824 > > > > I contacted MVAPICH2 team for this issue: http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2015-June/00562… > > > > They suggested me to contact MPICH team, as the issue is coming from higher level MPICH code. > > > > Can you please help me in resolving this. > > > > Regards, > > Chaitra > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Linux socket command may return 0 until MPI_Init_thread called.
by Jan Bierbaum 15 Jun '15

15 Jun '15

Hi Marvin! > Essentially, until you call MPI_Init_thread, the Linux socket command > will return 0 for all ranks except rank 0. This seems to me like a > problem. Indeed. 'socket' should not return 0. Quoting from its manpage: | On success, a file descriptor for the new socket is returned. On | error, -1 is returned, and errno is set appropriately. FD 0 is already taken by STDIN, so you should never get that return value unless you close STDIN first. > Given the following code sample... [...] > *The application returns output this with mpirun. Works for me: | $ mpirun -np 2 ./test-socket | Pre Socket: 10 | MPI_Init | Pre Socket: 6 | MPI_Init | Post Socket: 14 | Post Socket: 15 | | $ mpirun -np 3 ./test-socket | Pre Socket: 10 | MPI_Init | Pre Socket: 6 | MPI_Init | Pre Socket: 6 | MPI_Init | Post Socket: 14 | Post Socket: 15 | Post Socket: 10 > I am running this code sample using Red-Hat Enterprise Linux 7.1 with > mpich [...] > Version: 3.0.4 For me it's MPICH 3.1.4 on Debian 8 and I get similar results with MPICH 3.1rc on a local cluster. Taking into account the weird return value you get from 'socket' ... maybe your system is the problem. Regards, Jan _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0