June 2015 - discuss - mailman3.cels.anl.gov

Assertion error (enable-thread-cs=per-object)
by Guillem Blanco 15 Jun '15

15 Jun '15

Hi, I am using mpich-3.1.4 configured —enable-thread-cs=per-object. Every time I execute it I get the following assertion error: "Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 639: pkt->type >= 0 && pkt->type < MPIDI_CH3_PKT_END_ALL" However, if I compile mpich without —enable-thread-cs=per-object everything works correctly. Any help on this? Thanks, Guillem Blanco. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Application aborts at runtime
by Balaji, Pavan 15 Jun '15

15 Jun '15

It's the other way around — the MPI implementation is allowed to pick whatever upper bound it likes. The application is required to query this value before assigning the tag value. Even if MPICH increases this value, some other MPI implementation might not and you'll run into this problem again. The bottom line is that your application is not compliant with the MPI standard. You'll need to fix that. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji From: Chaitra Kumar Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" Date: Monday, June 15, 2015 at 12:13 AM To: "discuss(a)mpich.org<mailto:[email protected]>" Subject: Re: [mpich-discuss] Application aborts at runtime Hi Rajeev, Thanks for the reference to the relevant section of MPI 3.1 standard. As mentioned in the standard it specifies only the least value of upper bound which is 32767. An MPI implementation is free to set a high value of MPI_TAG_UB. So can this upper bound value (MPI_TAB_UB) be increased to allow higher tag values? Regards, Chaitra On Fri, Jun 12, 2015 at 9:31 AM, Thakur, Rajeev <thakur(a)mcs.anl.gov<mailto:[email protected]>> wrote: The tag cannot be larger than the value of the attribute key MPI_TAG_UB on MPI_COMM_WORLD. See Section 8.1.2, pg 334, of the MPI 3.1 standard. Rajeev > On Jun 11, 2015, at 8:17 PM, Chaitra Kumar <chaitragkumar(a)gmail.com<mailto:[email protected]>> wrote: > > Hi Junchao, > > Yes, the application has a requirement for big tag. > > With openMPI there are no issues while running the application. > > Can you please do the needful. > > Regards, > Chaitra > > On Thu, Jun 11, 2015 at 5:21 PM, Junchao Zhang <jczhang(a)mcs.anl.gov<mailto:[email protected]>> wrote: > As discussed in the mvapich mailing list, do you really need such big tag? > > --Junchao Zhang > > On Wed, Jun 10, 2015 at 8:35 PM, Chaitra Kumar <chaitragkumar(a)gmail.com<mailto:[email protected]>> wrote: > Hi Team, > > I am trying to run an application using MVAPICH2-2.0. The application compiles without any error, but at runtime, aborts with following error: > > Fatal error in PMPI_Isend:Invalid tag, error stack:PMPI_Isend(149): MPI_Isend(buf=0x7f59b5124300, count=8, MPI_UNSIGNED_LONG, dest=0, tag=1073741824, comm=0x84000001, request=0x132ae24) failed PMPI_Isend(98).: Invalid tag, value is 1073741824 > > I contacted MVAPICH2 team for this issue: http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2015-June/00562… > > They suggested me to contact MPICH team, as the issue is coming from higher level MPICH code. > > Can you please help me in resolving this. > > Regards, > Chaitra > > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Application aborts at runtime
by Chaitra Kumar 15 Jun '15

15 Jun '15

Hi Rajeev, Thanks for the reference to the relevant section of MPI 3.1 standard. As mentioned in the standard it specifies only the least value of upper bound which is 32767. An MPI implementation is free to set a high value of MPI_TAG_UB. So can this upper bound value (MPI_TAB_UB) be increased to allow higher tag values? Regards, Chaitra On Fri, Jun 12, 2015 at 9:31 AM, Thakur, Rajeev <thakur(a)mcs.anl.gov> wrote: > The tag cannot be larger than the value of the attribute key MPI_TAG_UB on > MPI_COMM_WORLD. See Section 8.1.2, pg 334, of the MPI 3.1 standard. > > Rajeev > > > On Jun 11, 2015, at 8:17 PM, Chaitra Kumar <chaitragkumar(a)gmail.com> > wrote: > > > > Hi Junchao, > > > > Yes, the application has a requirement for big tag. > > > > With openMPI there are no issues while running the application. > > > > Can you please do the needful. > > > > Regards, > > Chaitra > > > > On Thu, Jun 11, 2015 at 5:21 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> > wrote: > > As discussed in the mvapich mailing list, do you really need such big > tag? > > > > --Junchao Zhang > > > > On Wed, Jun 10, 2015 at 8:35 PM, Chaitra Kumar <chaitragkumar(a)gmail.com> > wrote: > > Hi Team, > > > > I am trying to run an application using MVAPICH2-2.0. The application > compiles without any error, but at runtime, aborts with following error: > > > > Fatal error in PMPI_Isend:Invalid tag, error stack:PMPI_Isend(149): > MPI_Isend(buf=0x7f59b5124300, count=8, MPI_UNSIGNED_LONG, dest=0, > tag=1073741824, comm=0x84000001, request=0x132ae24) failed PMPI_Isend(98).: > Invalid tag, value is 1073741824 > > > > I contacted MVAPICH2 team for this issue: > http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2015-June/00562… > > > > They suggested me to contact MPICH team, as the issue is coming from > higher level MPICH code. > > > > Can you please help me in resolving this. > > > > Regards, > > Chaitra > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

resource allocation and multiple mpi_comm_spawn's
by Arjen van Elteren 12 Jun '15

12 Jun '15

Hello, I'm working with an application that invokes multiple mpi_comm_spawn calls. I'm using mpiexec on a cluster without a resource manager or job queue, so plain ssh and fork calls and everything is managed by mpich. It looks like mpiexec (both hydra and mpd) re-use the hostfile from the start (and do not look at already allocated resources/used nodes). For example I have a hostfile like this: node01:1 node02:1 node03:1 When I run a call like this: MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, MPI_INFO_NULL, 2, MPI_COMM_SELF, &worker, MPI_ERRCODES_IGNORE); I get an allocation like this: node process --------------- ------------------ node01 manager node02 worker 1 node03 worker 2 Which is what I expected. But when I instead do 2 calls like this (i.e. each worker has one process, but there are 2 workers): MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, MPI_INFO_NULL, 1, MPI_COMM_SELF, &worker, MPI_ERRCODES_IGNORE); MPI_Comm_spawn(cmd, MPI_ARGV_NULL, number_of_workers, MPI_INFO_NULL, 1, MPI_COMM_SELF, &worker, MPI_ERRCODES_IGNORE); I get an allocation like this (both hydra and mpd): node process --------------- ------------------ node01 manager node02 worker 1 + worker 2 node03 Which is not what I expected at all! In fact, when I do this for a more complex example, I conclude that in MPI_Comm_spawn the hostfile is simply re-interpreted for every spawn and previous allocations in the same application are not accounted for. I know I could set hostname in the MPI_Comm_spawn call, but then I'm moving deployment information into my application (and I don't want to recompile or add a commandline argument for something that should be handled by mpiexec) Is there an option or easy fix for this problem? (I looked at the code of hydra, but I'm unsure how the different proxy's and processes divide this spawning work between them (I could not easily detect one "grand master" that does the allocation...) Kind regards, Arjen _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpich3.1.4 checkpoint restart failed error.
by Pankaj Karoriya 12 Jun '15

12 Jun '15

I am using mpich3.1.4 in os rhel 6.2 ,blcr-0.8.5.i have configure mpi with blcr library i configure mpich3.1.4 with following options; 1. ./configure --prefix=/usr/local/mpich-3.0.4-gcc --enable-cc --enable-fc --with-device=ch3:sock CC=gcc FC=gfortran --enable-checkpointing --with-hydra-ckpointlib=blcr --with-blcr=/usr/local/blcr-0.8.5 LD_LIBRARY_PATH=/usr/local/blcr-0.8.5/lib 2.make 3. make install After installation of mpi when running mpi checkpoint then its creates context file of mpi process. now i wants restart my mpi checkpont program than its give error is No alternate input file when restoring an external pipe kernel: blcr: cr_restore_all_files [32120]: Unable to restore fd 0 (type=4,err=-9) kernel: blcr: cr_rstrt_child [32120]: Unable to restore files! (err=-9) so whats i do for get of this problem.?? Thanks Pankaj _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Application aborts at runtime
by Thakur, Rajeev 11 Jun '15

11 Jun '15

The tag cannot be larger than the value of the attribute key MPI_TAG_UB on MPI_COMM_WORLD. See Section 8.1.2, pg 334, of the MPI 3.1 standard. Rajeev > On Jun 11, 2015, at 8:17 PM, Chaitra Kumar <chaitragkumar(a)gmail.com> wrote: > > Hi Junchao, > > Yes, the application has a requirement for big tag. > > With openMPI there are no issues while running the application. > > Can you please do the needful. > > Regards, > Chaitra > > On Thu, Jun 11, 2015 at 5:21 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > As discussed in the mvapich mailing list, do you really need such big tag? > > --Junchao Zhang > > On Wed, Jun 10, 2015 at 8:35 PM, Chaitra Kumar <chaitragkumar(a)gmail.com> wrote: > Hi Team, > > I am trying to run an application using MVAPICH2-2.0. The application compiles without any error, but at runtime, aborts with following error: > > Fatal error in PMPI_Isend:Invalid tag, error stack:PMPI_Isend(149): MPI_Isend(buf=0x7f59b5124300, count=8, MPI_UNSIGNED_LONG, dest=0, tag=1073741824, comm=0x84000001, request=0x132ae24) failed PMPI_Isend(98).: Invalid tag, value is 1073741824 > > I contacted MVAPICH2 team for this issue: http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2015-June/00562… > > They suggested me to contact MPICH team, as the issue is coming from higher level MPICH code. > > Can you please help me in resolving this. > > Regards, > Chaitra > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Application aborts at runtime
by Chaitra Kumar 11 Jun '15

11 Jun '15

Hi Junchao, Yes, the application has a requirement for big tag. With openMPI there are no issues while running the application. Can you please do the needful. Regards, Chaitra On Thu, Jun 11, 2015 at 5:21 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > As discussed in the mvapich mailing list, do you really need such big tag? > > --Junchao Zhang > > On Wed, Jun 10, 2015 at 8:35 PM, Chaitra Kumar <chaitragkumar(a)gmail.com> > wrote: > >> Hi Team, >> >> I am trying to run an application using MVAPICH2-2.0. The application >> compiles without any error, but at runtime, aborts with following error: >> >> Fatal error in PMPI_Isend:Invalid tag, error stack:PMPI_Isend(149): >> MPI_Isend(buf=0x7f59b5124300, count=8, MPI_UNSIGNED_LONG, dest=0, >> tag=1073741824, comm=0x84000001, request=0x132ae24) failed PMPI_Isend(98).: >> Invalid tag, value is 1073741824 >> >> I contacted MVAPICH2 team for this issue: >> http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2015-June/00562… >> >> They suggested me to contact MPICH team, as the issue is coming from >> higher level MPICH code. >> >> Can you please help me in resolving this. >> >> Regards, >> Chaitra >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Application aborts at runtime
by Junchao Zhang 11 Jun '15

11 Jun '15

As discussed in the mvapich mailing list, do you really need such big tag? --Junchao Zhang On Wed, Jun 10, 2015 at 8:35 PM, Chaitra Kumar <chaitragkumar(a)gmail.com> wrote: > Hi Team, > > I am trying to run an application using MVAPICH2-2.0. The application > compiles without any error, but at runtime, aborts with following error: > > Fatal error in PMPI_Isend:Invalid tag, error stack:PMPI_Isend(149): > MPI_Isend(buf=0x7f59b5124300, count=8, MPI_UNSIGNED_LONG, dest=0, > tag=1073741824, comm=0x84000001, request=0x132ae24) failed PMPI_Isend(98).: > Invalid tag, value is 1073741824 > > I contacted MVAPICH2 team for this issue: > http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2015-June/00562… > > They suggested me to contact MPICH team, as the issue is coming from > higher level MPICH code. > > Can you please help me in resolving this. > > Regards, > Chaitra > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Application aborts at runtime
by Chaitra Kumar 10 Jun '15

10 Jun '15

Hi Team, I am trying to run an application using MVAPICH2-2.0. The application compiles without any error, but at runtime, aborts with following error: Fatal error in PMPI_Isend:Invalid tag, error stack:PMPI_Isend(149): MPI_Isend(buf=0x7f59b5124300, count=8, MPI_UNSIGNED_LONG, dest=0, tag=1073741824, comm=0x84000001, request=0x132ae24) failed PMPI_Isend(98).: Invalid tag, value is 1073741824 I contacted MVAPICH2 team for this issue: http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2015-June/00562… They suggested me to contact MPICH team, as the issue is coming from higher level MPICH code. Can you please help me in resolving this. Regards, Chaitra _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Linux socket command may return 0 until MPI_Init_thread called.
by Marvin Smith 10 Jun '15

10 Jun '15

_______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0