March 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Failed to allocate memory for an unexpected message
by Luiz Carlos da Costa Junior 13 Mar '14

13 Mar '14

Thanks again Kenneth, I could solve using MPI_TESTANY. Regards, Luiz On 13 March 2014 15:35, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > On 03/13/2014 12:35 PM, Luiz Carlos da Costa Junior wrote: > >> Does anyone have any clue about this? >> >> Thanks in advance. >> >> >> On 12 March 2014 14:40, Luiz Carlos da Costa Junior <lcjunior(a)ufrj.br >> <mailto:[email protected]>> wrote: >> >> Dear Kenneth, >> >> Thanks for your quick reply. >> I tested your suggestion and, unfortunately, this approach didn't >> work. >> >> Question: when I call MPI_IPROBE it accounts also for the messages >> that were already received asynchronously? >> > > That should not be the case. If a message has been matched by a > recv/irecv, MPI_Probe should not match it again. > > > >> Is there any way to know, for my list of mpi_requests (from my >> MPI_IRECV's), which ones are "opened" and which ones have messages? >> > > MPI_Test will take a request as an argument and tell you whether or not > that requested operation has been completed. > > Ken > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Failed to allocate memory for an unexpected message
by Kenneth Raffenetti 13 Mar '14

13 Mar '14

On 03/13/2014 12:35 PM, Luiz Carlos da Costa Junior wrote: > Does anyone have any clue about this? > > Thanks in advance. > > > On 12 March 2014 14:40, Luiz Carlos da Costa Junior <lcjunior(a)ufrj.br > <mailto:[email protected]>> wrote: > > Dear Kenneth, > > Thanks for your quick reply. > I tested your suggestion and, unfortunately, this approach didn't work. > > Question: when I call MPI_IPROBE it accounts also for the messages > that were already received asynchronously? That should not be the case. If a message has been matched by a recv/irecv, MPI_Probe should not match it again. > > Is there any way to know, for my list of mpi_requests (from my > MPI_IRECV's), which ones are "opened" and which ones have messages? MPI_Test will take a request as an argument and tell you whether or not that requested operation has been completed. Ken

1 0

Re: [mpich-discuss] Failed to allocate memory for an unexpected message
by Luiz Carlos da Costa Junior 13 Mar '14

13 Mar '14

Does anyone have any clue about this? Thanks in advance. On 12 March 2014 14:40, Luiz Carlos da Costa Junior <lcjunior(a)ufrj.br>wrote: > Dear Kenneth, > > Thanks for your quick reply. > I tested your suggestion and, unfortunately, this approach didn't work. > > Question: when I call MPI_IPROBE it accounts also for the messages that > were already received asynchronously? > > Is there any way to know, for my list of mpi_requests (from my > MPI_IRECV's), which ones are "opened" and which ones have messages? > > Regards, > > > On 11 March 2014 17:00, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > >> You could use MPI_Probe/MPI_Iprobe and pass in your "data" tag to test >> for any more pending messages. >> >> Ken >> >> >> On 03/11/2014 02:50 PM, Luiz Carlos da Costa Junior wrote: >> >>> Dear all, >>> >>> I am all set with your suggestions and my program is working quite well >>> without any "unexpected message" since then, thanks again. >>> >>> However now I am facing a small problem I describe next. >>> >>> My receiver process receives actually 2 types of messages (2 different >>> tags): >>> >>> * the basic tag means that the message is a "data message" ("data" >>> >>> tag) that should be processed. >>> * the second one means that the worker process is done and it will >>> >>> send no more messages ("end_of_processing" tag). >>> >>> Once all worker processes send their end_of_processing tag, the receiver >>> process finishes its execution. >>> >>> The problem I noticed is that some of the last messages sent by the >>> worker processes were not being processed. I think the problem is >>> related with the logic I am using with MPI_WAITANY in the receiver >>> process. I am simply counting the number of end_of_processing messages >>> received and if it reaches the number of worker processes, I finish the >>> execution without checking if there are more messages to be received at >>> the MPI_WAITANY queue. >>> >>> As the order that messages arrive is not relevant for MPI_WAITANY I >>> think that my logic forgets some of the messages at the end of the >>> queue. Is this right? >>> >>> Is there any way to check if there is any pending request to be >>> processed? >>> >>> Best regards, >>> Luiz >>> >>> >>> On 16 January 2014 16:57, "Antonio J. Peña" <apenya(a)mcs.anl.gov >>> <mailto:[email protected]>> wrote: >>> >>> >>> A profiling of both of your codes would help to understand where the >>> time is spent and the difference between them in terms of >>> performance. >>> >>> Antonio >>> >>> >>> >>> On 01/16/2014 12:47 PM, Luiz Carlos da Costa Junior wrote: >>> >>>> Yes, I am comparing original x new implementation. >>>> >>>> The original implementation is as follows. >>>> >>>> c----------------------------------------------------------- >>>> ------------ >>>> subroutine my_receiver_original >>>> c ------------------------------------------------------------ >>>> ------ >>>> (...) >>>> >>>> c Local >>>> c ----- >>>> integer*4 m_stat(MPI_STATUS_SIZE) >>>> character card*(zbuf) ! buffer for messages received >>>> >>>> do while( keep_receiving ) >>>> call MPI_RECV(card, zbuf, MPI_CHARACTER, >>>> . MPI_ANY_SOURCE, M_RECCSV, MY_COMM, >>>> . m_stat, m_ierr ) >>>> >>>> c Process message: disk IO >>>> c --------------- >>>> <DO SOMETHING> >>>> if( SOMETHING_ELSE ) then >>>> keep_receiving = .false. >>>> end if >>>> end do >>>> >>>> (...) >>>> >>>> return >>>> end >>>> >>>> Regards, >>>> Luiz >>>> >>>> On 16 January 2014 16:19, Balaji, Pavan <balaji(a)mcs.anl.gov >>>> <mailto:[email protected]>> wrote: >>>> >>>> >>>> On Jan 16, 2014, at 12:16 PM, Luiz Carlos da Costa Junior >>>> <lcjunior(a)ufrj.br <mailto:[email protected]>> wrote: >>>> > No, these failures don't occur all the time. I have a >>>> successful run (with my original implementation) which I am >>>> using as the base case for comparison. >>>> >>>> What are the two cases you are comparing? Original >>>> implementation vs. new implementation? What's the original >>>> implementation? >>>> >>>> -- Pavan >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org <mailto: >>>> discuss(a)mpich.org> >>>> >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >>>> >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >>> >>> -- >>> Antonio J. Peña >>> Postdoctoral Appointee >>> Mathematics and Computer Science Division >>> Argonne National Laboratory >>> 9700 South Cass Avenue, Bldg. 240, Of. 3148 >>> Argonne, IL 60439-4847 >>> apenya(a)mcs.anl.gov <mailto:[email protected]> >>> www.mcs.anl.gov/~apenya <http://www.mcs.anl.gov/~apenya> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > >

1 0

Re: [mpich-discuss] Value of MPI_TAG_UB
by Bibrak Qamar 13 Mar '14

13 Mar '14

Thanks Pavan ! Bibrak Qamar On Thu, Mar 13, 2014 at 3:54 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Yes, the tag UB is correct. MPICH reserves one of the tag bits to help > with internal fault information propagation. > > -- Pavan > > On Mar 13, 2014, at 1:30 AM, Bibrak Qamar <bibrakc(a)gmail.com> wrote: > > > I am trying to get the value of MPI_TAG_UB using the following code: > > > > MPI_Aint * maxTag;; > > int error, flag; > > > > error = MPI_Comm_get_attr(MPI_COMM_WORLD, MPI_TAG_UB, &maxTag, &flag); > > > > if(error != MPI_SUCCESS || !flag ){ > > printf( " ERROR IN native mpitagub\n"); > > } > > > > printf( "mpitagub = %d\n", *maxTag); > > > > > > For MPICH3 I get > > > > mpitagub = 1073741823 > > > > I also did it for OpenMPI and I get > > > > mpitagub = 2147483647 > > > > The value from OpenMPI makes sense since its 2^31 -1. But at the end the > MPI_TAG_UB is MPI implementation dependent. > > > > So I was wondering the value that I got from MPICH3 (mpitagub = > 1073741823) is this OK? or my code is wrong somewhere? > > > > Thanks > > Bibrak > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Value of MPI_TAG_UB
by Balaji, Pavan 13 Mar '14

13 Mar '14

Yes, the tag UB is correct. MPICH reserves one of the tag bits to help with internal fault information propagation. — Pavan On Mar 13, 2014, at 1:30 AM, Bibrak Qamar <bibrakc(a)gmail.com> wrote: > I am trying to get the value of MPI_TAG_UB using the following code: > > MPI_Aint * maxTag;; > int error, flag; > > error = MPI_Comm_get_attr(MPI_COMM_WORLD, MPI_TAG_UB, &maxTag, &flag); > > if(error != MPI_SUCCESS || !flag ){ > printf( " ERROR IN native mpitagub\n"); > } > > printf( "mpitagub = %d\n", *maxTag); > > > For MPICH3 I get > > mpitagub = 1073741823 > > I also did it for OpenMPI and I get > > mpitagub = 2147483647 > > The value from OpenMPI makes sense since its 2^31 -1. But at the end the MPI_TAG_UB is MPI implementation dependent. > > So I was wondering the value that I got from MPICH3 (mpitagub = 1073741823) is this OK? or my code is wrong somewhere? > > Thanks > Bibrak > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Value of MPI_TAG_UB
by Bibrak Qamar 13 Mar '14

13 Mar '14

I am trying to get the value of MPI_TAG_UB using the following code: MPI_Aint * maxTag;; int error, flag; error = MPI_Comm_get_attr(MPI_COMM_WORLD, MPI_TAG_UB, &maxTag, &flag); if(error != MPI_SUCCESS || !flag ){ printf( " ERROR IN native mpitagub\n"); } printf( "mpitagub = %d\n", *maxTag); For MPICH3 I get mpitagub = 1073741823 I also did it for OpenMPI and I get mpitagub = 2147483647 The value from OpenMPI makes sense since its 2^31 -1. But at the end the MPI_TAG_UB is MPI implementation dependent. *So I was wondering the value that I got from MPICH3 (mpitagub = 1073741823) is this OK? or my code is wrong somewhere?* Thanks Bibrak

1 0

Re: [mpich-discuss] Increasing MPI ranks
by Jed Brown 12 Mar '14

12 Mar '14

Jeffrey Larson <jmlarson(a)anl.gov> writes: > I am not calling the cpi.py script directly. The master is spawning those > processes. So I call > > $ mpiexec -n 30 python master.py > > Then each of the 30 ranks should spawn a cpi.py process. But with the > attached master.py and cpi.py (directly from the mpi4py tutorial), you can > see the errors I get: > > [jlarson@mintthinkpad tutorial_example]$ mpiexec -n 30 python master.py > [mpiexec@mintthinkpad] control_cb (pm/pmiserv/pmiserv_cb.c:200): assert > (!closed) failed > [mpiexec@mintthinkpad] HYDT_dmxu_poll_wait_for_event > (tools/demux/demux_poll.c:76): callback returned error status > [mpiexec@mintthinkpad] HYD_pmci_wait_for_completion > (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > [mpiexec@mintthinkpad] main (ui/mpich/mpiexec.c:336): process manager error > waiting for completion > > As was previously stated, this appears to be an mpi4py problem and not a > mpich question. The problem seems to be related to passing arguments. This works for me: diff --git a/master.py b/master.py index 620c484..e103356 100755 --- a/master.py +++ b/master.py @@ -3,7 +3,7 @@ from mpi4py import MPI import numpy import sys -comm = MPI.COMM_SELF.Spawn(sys.executable, args=['cpi.py'], maxprocs=5) +comm = MPI.COMM_SELF.Spawn('./cpi.py', None, maxprocs=5) N = numpy.array(100, 'i') comm.Bcast([N, MPI.INT] root=MPI.ROOT) The original also works fine with Open MPI. I managed to reproduce with pure C and MPICH. diff --git i/demo/spawning/cpi-master.c w/demo/spawning/cpi-master.c index 7bff4e3..6f8715b 100644 --- i/demo/spawning/cpi-master.c +++ w/demo/spawning/cpi-master.c @@ -6,6 +6,7 @@ int main(int argc, char *argv[]) { char cmd[32] = "cpi-worker-c.exe"; + char *wargs[] = {"a",NULL}; MPI_Comm worker; int n; double pi; @@ -15,7 +16,7 @@ int main(int argc, char *argv[]) if (argc > 1) strcpy(cmd, argv[1]); printf("%s -> %s\n", argv[0], cmd); - MPI_Comm_spawn(cmd, MPI_ARGV_NULL, 5, + MPI_Comm_spawn(cmd, wargs, 5, MPI_INFO_NULL, 0, MPI_COMM_SELF, &worker, MPI_ERRCODES_IGNORE); The attached includes a sleep(1) in the client to make the crash more reproducible. $ make CC=/opt/mpich/bin/mpicc cpi-master cpi-worker /opt/mpich/bin/mpicc cpi-master.c -o cpi-master /opt/mpich/bin/mpicc cpi-worker.c -o cpi-worker $ /opt/mpich/bin/mpiexec -n 30 ./cpi-master ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker ./cpi-master -> ./cpi-worker Assertion failed in file ../src/mpi/coll/helper_fns.c at line 491: status->MPI_TAG == recvtag internal ABORT - process 0 Assertion failed in file ../src/mpi/coll/helper_fns.c at line 491: status->MPI_TAG == recvtag internal ABORT - process 0 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 7329 RUNNING AT batura = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:27:0@batura] HYD_pmcd_pmip_control_cmd_cb (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:27:0@batura] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:27:0@batura] main (../../../../src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [proxy:28:0@batura] HYD_pmcd_pmip_control_cmd_cb (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:28:0@batura] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:28:0@batura] main (../../../../src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [proxy:29:0@batura] HYD_pmcd_pmip_control_cmd_cb (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:29:0@batura] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:29:0@batura] main (../../../../src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@batura] HYDT_bscu_wait_for_completion (../../../../src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@batura] HYDT_bsci_wait_for_completion (../../../../src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@batura] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@batura] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:336): process manager error waiting for completion > Since you are curious about the application, I the motivating example > involves the numerical optimization of the output from an expensive > simulation. I do not have access to the simulation code, so my master will > tell the workers where they need to evaluate the expensive simulation. Then > the simulation might itself depend heavily on MPI. How will the code you don't "have access to" report back? Presumably it was not written to use MPI_Comm_get_parent? When possible, it's better for the simulation to take an MPI_Comm argument on which to run.

1 0

Re: [mpich-discuss] Increasing MPI ranks
by Jeffrey Larson 12 Mar '14

12 Mar '14

I am not calling the cpi.py script directly. The master is spawning those processes. So I call $ mpiexec -n 30 python master.py Then each of the 30 ranks should spawn a cpi.py process. But with the attached master.py and cpi.py (directly from the mpi4py tutorial), you can see the errors I get: [jlarson@mintthinkpad tutorial_example]$ mpiexec -n 30 python master.py [mpiexec@mintthinkpad] control_cb (pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed [mpiexec@mintthinkpad] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec@mintthinkpad] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@mintthinkpad] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion As was previously stated, this appears to be an mpi4py problem and not a mpich question. Since you are curious about the application, I the motivating example involves the numerical optimization of the output from an expensive simulation. I do not have access to the simulation code, so my master will tell the workers where they need to evaluate the expensive simulation. Then the simulation might itself depend heavily on MPI. But I welcome your input on the design paradigm to avoid the "sharp edges" Thank you again, Jeff On Wed, Mar 12, 2014 at 5:15 PM, Jed Brown <jed(a)jedbrown.org> wrote: > Jeffrey Larson <jmlarson(a)anl.gov> writes: > > > I am trying to have a single master, with a group of workers who are > > themselves calculating function values. The evaluation of the function > may > > itself involve spawning MPI tasks. > > How are you running the cpi test then? I ran it with many spawned > processes > > MPI.COMM_SELF.Spawn(cmd, None, 30) > > and with one spawned process on each of many masters. Neither crashed > with current MPICH or Open MPI. What exactly is needed to reproduce the > failure you see? > > > I am curious why you want all this process spawning (there are lots of > sharp edges to deploying this approach), but mpi4py should work. >

1 0

Re: [mpich-discuss] Increasing MPI ranks
by Jed Brown 12 Mar '14

12 Mar '14

Jeffrey Larson <jmlarson(a)anl.gov> writes: > I am trying to have a single master, with a group of workers who are > themselves calculating function values. The evaluation of the function may > itself involve spawning MPI tasks. How are you running the cpi test then? I ran it with many spawned processes MPI.COMM_SELF.Spawn(cmd, None, 30) and with one spawned process on each of many masters. Neither crashed with current MPICH or Open MPI. What exactly is needed to reproduce the failure you see? I am curious why you want all this process spawning (there are lots of sharp edges to deploying this approach), but mpi4py should work.

1 0

Re: [mpich-discuss] MPI_Address()
by Balaji, Pavan 12 Mar '14

12 Mar '14

On Mar 12, 2014, at 1:15 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > 1. ./configure --enable|disable-mpi3-deleted-interfaces How about --enable-deleted-interfaces=[level] level can be “mpi3” for now, but in the future, “mpi4” or “mpi27” could be an option as well. This will get messy if a future MPI standard decides to add back some of the older routines, but I don’t see that happening in the near future (or ever). > 2. MPI_T name: mpi3_deleted_interfaces_check, which can take one of four values: none, warn_once, warn, or error Our MPI_T variables are of the type MPIR_CVAR_*. The values seem fine to me. In our setup, all CVARs will have an equivalent environment variable to control them as well. — Pavan

1 0