June 2017 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] bind/listen race condition
by Kenneth Raffenetti 29 Jun '17

29 Jun '17

Hi Steve, Thanks for the report. I was able to reproduce it on my laptop with the latest git revision. I've created a Github issue to track it. Hopefully it will be fixed soon. https://github.com/pmodels/mpich/issues/2665 Ken On 06/27/2017 02:11 PM, Steve Krueger wrote: > I'm using a rather old 1.4.1p1 version, but I checked the latest sources, and I believe the > problem still exists. > > When starting multiple MPICH jobs, and using an MPICH_PORT_RANGE, I occasionally see the error: > HYDU_sock_listen (./utils/sock/sock.c:128): listen error (Address already in use) > > I believe two processes are successfully bind()ing the same port in the range, but then the second listen() call returns the error. > > The code in sock.c loops through the port range attempting to find a port to bind to, and once the bind() succeeds, it only call listen() once. I think the code should tolerate an EADDRINUSE error from listen() and retry a new port. > > The problem is easier to reproduce if you add a sleep(10) between the bind() and listen(), and > then run two mpi jobs with the same MPICH_PORT_RANGE. > > sk > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

bind/listen race condition
by Steve Krueger 27 Jun '17

27 Jun '17

I'm using a rather old 1.4.1p1 version, but I checked the latest sources, and I believe the problem still exists. When starting multiple MPICH jobs, and using an MPICH_PORT_RANGE, I occasionally see the error: HYDU_sock_listen (./utils/sock/sock.c:128): listen error (Address already in use) I believe two processes are successfully bind()ing the same port in the range, but then the second listen() call returns the error. The code in sock.c loops through the port range attempting to find a port to bind to, and once the bind() succeeds, it only call listen() once. I think the code should tolerate an EADDRINUSE error from listen() and retry a new port. The problem is easier to reproduce if you add a sleep(10) between the bind() and listen(), and then run two mpi jobs with the same MPICH_PORT_RANGE. sk _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec ignoring host setting in configfile
by Kenneth Raffenetti 23 Jun '17

23 Jun '17

On 06/23/2017 10:19 AM, Alexander Rast wrote: > What I need at this point is a better understanding of the > implementation philosophy of MPICH and Hydra because I fear that without > it I'm going to run into a series of endless surprises that will > generate questions that might never have arisen if I'd known what the > internal thinking of the MPICH group was from the outset. Forgive me if > some of what I say or ask for here is quite general, but I'm trying to > put MPICH in enough context that I can reasonably predict how it will > behave in various scenarios. > > I'm giving you my own thinking here so you have some context on where > I'm coming from, not as an expectation or demand that things should work > this way but as a description of what I'd imagine in the absence of any > better information. It's going to be rather long and I apologise for > that but I think you might need it in order to make sense of what I'm > saying. > > Also posting this to the mailing list, because if productive information > comes out of this exchange it may be useful for others in future as a > sort of supplementary documentation. > > On Thu, Jun 22, 2017 at 6:57 PM, Kenneth Raffenetti > <raffenet(a)mcs.anl.gov <mailto:[email protected]>> wrote: > > On 06/22/2017 11:16 AM, Alexander Rast wrote: > > On Thu, Jun 22, 2017 at 4:14 PM, Kenneth Raffenetti > <raffenet(a)mcs.anl.gov <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > -host(s) is a global option. You can only specify it once. > > > ? This doesn't make sense to me. Example 8.13 from the MPI > specification has: > > mpiexec myprog infile1 : myprog infile2 : myprog infile3, > corresponding to Form A > > mpiexec {<above arguments>} : {...} : {...} : ... : { ...} > > which invokes potentially multiple programs with different > arguments (of which host is one) > > and then mentions that Form B: mpiexec -configfile <filename> > has lines of the form separated by the colons in Form A. > > So there's no reason, I would think, that a Form B file can't > have multiple lines with different hosts specified for the same > executable. > > > The reason is that Hydra does not support it. You are allowed one > -host(s) argument. If you want to submit a patch to allow more than > one instance, we will consider it. > > > I have to concede that this comes as a surprise. I didn't see anything > previously that said only one -host argument is allowed, and if I read > through the MPI specification on mpiexec and the definition of the -host > argument I would definitely come to the opposite conclusion. It seemed > clear that -host would be allowed multiple times for multiple > specifications. Now, the specification of mpiexec is optional but as I > understood it if mpiexec is supported then the option settings described > in the specification should operate like an MPI_COMM_SPAWN command. From > my point of view it would have been surprising and unexpected behaviour > for -host to behave differently on the first invocation of > MPI_COMM_SPAWN than it did on subsequent invocations. A given > implementation would be free to choose not to support -host as an option > switch, but if this were the case, I'd expect it to raise an error or at > least a warning rather than silently ignoring the switch. The particular > interpretation MPICH seems to have taken as to how to handle this switch > is one that would never have occurred to me under any circumstances, and > as you can see, would never have occurred to me *even after the fact*, > without explicit indication that this was the path chosen, at which > point as you can also see I would still have responded with utter > astonishment. I don't understand the rationale. > > MPICH supports a different option: a hosts file specified using -f, that > much I understood. The MPI specification says that mpiexec may support > any number of additional implementation-dependent options, whose > behaviour is entirely dependent upon the implementation, but that would > simply imply to me that there is an alternative way to specify hosts, > users, process mappings, etc. Likewise a -hosts option is possible but I > would have considered that an entirely separate case (with possibly > different behaviour) from -host: just because 2 switches look similar > doesn't mean they necessarily are the same. With information now > available I am given to understand that this is not the case but it is > certainly not what I would have expected a priori. > > One of the problems is that MPI documentation itself is very complete > but documentation on implementations is very thin, MPICH is not alone in > this aspect, and a lot of the details of MPI are > implementation-dependent. This leaves the user struggling with best > guesses as to what an implementation will do. > > Use of a hosts file carries with it its own ambiguities. As I understood > mpiexec as described in the specification for MPI, the -n switch only > corresponds to a request to start that many processes; it doesn't come > with any guarantees. If everything is working according to plan then I > would expect n processes will be started, but I wasn't ready to count on > that as a firm guarantee. This means that the behaviour of the system > when either fewer processes than asked for could be started, or when the > host-processes mapping couldn't be satisfied, is unclear. For instance, > if I had -n 10 specified and a hosts file with A:3 B:2 C:3 D:2 then > suppose in actual fact only 7 processes could be started? Would it have > been safe to assume that the mapping in actual fact were A: 0-2, B: 3-4, > C: 5-6?. That's one possibility but not the only one. Or what if for > some reason A were only able to run 2 processes? Would the system remap > so that 10 were supported, e.g. A: 0-1, B: 2-4, C: 5-6, D: 7-9? Or some > other combination? Or fail altogether? Maybe the combinations are > manageable in small configurations but if you had 1000 hosts supporting > some number of processes between 0 and 100, for an application > distributing a total of 25,000 processes, the possible combinations > become hopeless to calculate. Internally Hydra may have a deterministic > assignment algorithm, but without delving deeply into the internals this > wouldn't be easy to determine. > > At any rate, from the point of view of a user, I'd have considered a > fixed mapping algorithm, where one were at the mercy of what Hydra or > any other process manager decided to do in terms of which process ended > up where, to be an exceptionally onerous restriction. It would be from > my point of view sort of fundamental that the user be able to specify if > they want to exactly which process maps to which hardware resource - > after all, there's no way an external process manager can know the > details of what hardware capabilities may be at each node and there may > be very strong reasons why the mapping needs to be as the user wants. > This is one of the main reasons I'd have expected that if the -host > option were even allowed it would be usable in a line-by-line (or > colon-separated arglist) mode. > > It looks as if by a combination of hosts file and mpiexec options you > can *probably* achieve something similar but this is frightfully > implicit. On a purely personal level I find very implicit specifications > - where a resultant configuration can be *deduced* from the combination > of inputs to the configuration generator, but isn't already explicitly > plain in the inputs themselves - to be confusing and difficult to work > with. In this case you have a result which is dependent not merely on > the options passed to mpiexec and the lines of the hosts file, but on > the order of lines within the hosts file and the order of executable > specifications in the mpiexec invocation. As noted I also have concerns > about what happens if the mapping can't be achieved exactly as the > implicit specification suggests but would be in principle feasible with > a different mapping. > > So what I'm looking for, again, is some background on the implementation > and expected usage model for MPICH in the sense not of an empirical 'do > this because that's what works' but rather a more top-down 'we imagine > problems cast in this form, and so have configured MPICH to work like > this' so that all these choices can be put into a context I can make > sense of. Maybe you're not the best placed to answer this sort of > question. Can someone else in the discussion list provide any insights? If you are looking for background on expected usage of MPI(CH), you may find this book useful: https://mitpress.mit.edu/using-MPI-3ed As for why Hydra was implemented the way it is, I'll only add that features are implemented based on common practice and user demand. What you are asking for is neither, from my experience. > > > > Further, your configfile does not make much sense since > your are > using the same program on both lines. You could rather just do: > > mpiexec -n 4 -ppn 2 -hosts Shakespeare,Burns ./mpi_io_test > > > I have not seen the -ppn switch listed anywhere. What does this > do? Is this intended to be a 'limit number of processes to n on > the hosts specified? > > > $ mpiexec -h | grep ppn > -ppn processes per node > > Notwithstanding this isn't what is needed anyway, because what > if I want specifically to run at most 2 processes on Shakespeare > but 5 on Burns (imagine, for example, that Shakespeare is > already loaded with other processes, so I need to limit it for > the executable in concern)? > > > Irregular mappings like this can be done with a hostfile. > > > In any case the point is, using the same program on both lines > to me seems straightforward enough given that there are numerous > instances (as ex. 8.13 above) where various switches and options > might be different. > > I note also that -hosts is different from the -host switch > listed as pre-defined for mpiexec. > > > Hydra supports both -host and -hosts. They mean the same thing. > > > In my case we have a system where the objective is to direct > specific instances of an application or possibly several to > specific hosts - so that the mapping can be fixed. In the limit > this might consist of -n 1 -host <x> <executable> lines, each > directing a specific instance of the executable to a specific host. > > > Again, this will not work with today's Hydra. I would suggest > playing around some more with the available options to see if > another solution might work for your use-case. > > > Back to the real situation. What we (will) have is a machine with a > large number of compute nodes. Simply by force of numbers these compute > nodes are going to be unreliable: some of them will work, some not, > although we do assume that at actual run-time the list of working nodes > can be gained by discovery from a root process and configured as a > static mapping for the purposes of any given run. The critical point > though is that it's only at system initialisation time, not before, that > this information can be known. A root process will take a problem (i.e. > an application) specification and determine a mapping for the > application based on the available hardware topology, the problem > topology, and the predicted patterns of use. It then will need to run an > MPI_COMM_SPAWN_MULTIPLE to distribute the problem to the working nodes, > and we need to make sure the distribution is exactly as specified or the > system simply won't work - problem components will be distributed > differently from the expected topology, large chunks of data might end > up in entirely unexpected places and the application will fall down > almost immediately (deadlock is very likely as well). The main thing I > need to understand is exactly what the MPI_COMM_SPAWN_MULTIPLE will do. > As you can see there are a lot of secondary questions related to things > like where I/O goes, what user is on which host, Fortran support etc. > but these are only sub-problems of the main issue which is getting to > grips with MPI_COMM_SPAWN_MULTIPLE. > > As you may be able to see, using a hosts file would be awkward at best, > because we're not going to know until application startup what the hosts > would actuallly be. The best solution I can come up with if it really is > the case that this is the only way to map processes to hosts is to have > a 2-stage program that does discovery and then pipes the result into a > second stage invoked through mpiexec but I'll need to go away and see if > there are any interactions with MPI_COMM_SPAWN like what I found related > to different users to know if this is going to work. I'll run some tests > and let you know what I find This is more of an application development issue, and not really in the scope of this list. Nevertheless, have you considered allowing processes to determine their computational role *after* initialization rather than launching new binaries? Ken _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec ignoring host setting in configfile
by Alexander Rast 23 Jun '17

23 Jun '17

What I need at this point is a better understanding of the implementation philosophy of MPICH and Hydra because I fear that without it I'm going to run into a series of endless surprises that will generate questions that might never have arisen if I'd known what the internal thinking of the MPICH group was from the outset. Forgive me if some of what I say or ask for here is quite general, but I'm trying to put MPICH in enough context that I can reasonably predict how it will behave in various scenarios. I'm giving you my own thinking here so you have some context on where I'm coming from, not as an expectation or demand that things should work this way but as a description of what I'd imagine in the absence of any better information. It's going to be rather long and I apologise for that but I think you might need it in order to make sense of what I'm saying. Also posting this to the mailing list, because if productive information comes out of this exchange it may be useful for others in future as a sort of supplementary documentation. On Thu, Jun 22, 2017 at 6:57 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > On 06/22/2017 11:16 AM, Alexander Rast wrote: > >> On Thu, Jun 22, 2017 at 4:14 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov >> <mailto:[email protected]>> wrote: >> >> -host(s) is a global option. You can only specify it once. >> >> >> ? This doesn't make sense to me. Example 8.13 from the MPI specification >> has: >> >> mpiexec myprog infile1 : myprog infile2 : myprog infile3, corresponding >> to Form A >> >> mpiexec {<above arguments>} : {...} : {...} : ... : { ...} >> >> which invokes potentially multiple programs with different arguments (of >> which host is one) >> >> and then mentions that Form B: mpiexec -configfile <filename> has lines >> of the form separated by the colons in Form A. >> >> So there's no reason, I would think, that a Form B file can't have >> multiple lines with different hosts specified for the same executable. >> > > The reason is that Hydra does not support it. You are allowed one -host(s) > argument. If you want to submit a patch to allow more than one instance, we > will consider it. > I have to concede that this comes as a surprise. I didn't see anything previously that said only one -host argument is allowed, and if I read through the MPI specification on mpiexec and the definition of the -host argument I would definitely come to the opposite conclusion. It seemed clear that -host would be allowed multiple times for multiple specifications. Now, the specification of mpiexec is optional but as I understood it if mpiexec is supported then the option settings described in the specification should operate like an MPI_COMM_SPAWN command. From my point of view it would have been surprising and unexpected behaviour for -host to behave differently on the first invocation of MPI_COMM_SPAWN than it did on subsequent invocations. A given implementation would be free to choose not to support -host as an option switch, but if this were the case, I'd expect it to raise an error or at least a warning rather than silently ignoring the switch. The particular interpretation MPICH seems to have taken as to how to handle this switch is one that would never have occurred to me under any circumstances, and as you can see, would never have occurred to me *even after the fact*, without explicit indication that this was the path chosen, at which point as you can also see I would still have responded with utter astonishment. I don't understand the rationale. MPICH supports a different option: a hosts file specified using -f, that much I understood. The MPI specification says that mpiexec may support any number of additional implementation-dependent options, whose behaviour is entirely dependent upon the implementation, but that would simply imply to me that there is an alternative way to specify hosts, users, process mappings, etc. Likewise a -hosts option is possible but I would have considered that an entirely separate case (with possibly different behaviour) from -host: just because 2 switches look similar doesn't mean they necessarily are the same. With information now available I am given to understand that this is not the case but it is certainly not what I would have expected a priori. One of the problems is that MPI documentation itself is very complete but documentation on implementations is very thin, MPICH is not alone in this aspect, and a lot of the details of MPI are implementation-dependent. This leaves the user struggling with best guesses as to what an implementation will do. Use of a hosts file carries with it its own ambiguities. As I understood mpiexec as described in the specification for MPI, the -n switch only corresponds to a request to start that many processes; it doesn't come with any guarantees. If everything is working according to plan then I would expect n processes will be started, but I wasn't ready to count on that as a firm guarantee. This means that the behaviour of the system when either fewer processes than asked for could be started, or when the host-processes mapping couldn't be satisfied, is unclear. For instance, if I had -n 10 specified and a hosts file with A:3 B:2 C:3 D:2 then suppose in actual fact only 7 processes could be started? Would it have been safe to assume that the mapping in actual fact were A: 0-2, B: 3-4, C: 5-6?. That's one possibility but not the only one. Or what if for some reason A were only able to run 2 processes? Would the system remap so that 10 were supported, e.g. A: 0-1, B: 2-4, C: 5-6, D: 7-9? Or some other combination? Or fail altogether? Maybe the combinations are manageable in small configurations but if you had 1000 hosts supporting some number of processes between 0 and 100, for an application distributing a total of 25,000 processes, the possible combinations become hopeless to calculate. Internally Hydra may have a deterministic assignment algorithm, but without delving deeply into the internals this wouldn't be easy to determine. At any rate, from the point of view of a user, I'd have considered a fixed mapping algorithm, where one were at the mercy of what Hydra or any other process manager decided to do in terms of which process ended up where, to be an exceptionally onerous restriction. It would be from my point of view sort of fundamental that the user be able to specify if they want to exactly which process maps to which hardware resource - after all, there's no way an external process manager can know the details of what hardware capabilities may be at each node and there may be very strong reasons why the mapping needs to be as the user wants. This is one of the main reasons I'd have expected that if the -host option were even allowed it would be usable in a line-by-line (or colon-separated arglist) mode. It looks as if by a combination of hosts file and mpiexec options you can *probably* achieve something similar but this is frightfully implicit. On a purely personal level I find very implicit specifications - where a resultant configuration can be *deduced* from the combination of inputs to the configuration generator, but isn't already explicitly plain in the inputs themselves - to be confusing and difficult to work with. In this case you have a result which is dependent not merely on the options passed to mpiexec and the lines of the hosts file, but on the order of lines within the hosts file and the order of executable specifications in the mpiexec invocation. As noted I also have concerns about what happens if the mapping can't be achieved exactly as the implicit specification suggests but would be in principle feasible with a different mapping. So what I'm looking for, again, is some background on the implementation and expected usage model for MPICH in the sense not of an empirical 'do this because that's what works' but rather a more top-down 'we imagine problems cast in this form, and so have configured MPICH to work like this' so that all these choices can be put into a context I can make sense of. Maybe you're not the best placed to answer this sort of question. Can someone else in the discussion list provide any insights? > > >> Further, your configfile does not make much sense since your are >> using the same program on both lines. You could rather just do: >> >> mpiexec -n 4 -ppn 2 -hosts Shakespeare,Burns ./mpi_io_test >> >> >> I have not seen the -ppn switch listed anywhere. What does this do? Is >> this intended to be a 'limit number of processes to n on the hosts >> specified? >> > > $ mpiexec -h | grep ppn > -ppn processes per node > > Notwithstanding this isn't what is needed anyway, because what if I want >> specifically to run at most 2 processes on Shakespeare but 5 on Burns >> (imagine, for example, that Shakespeare is already loaded with other >> processes, so I need to limit it for the executable in concern)? >> > > Irregular mappings like this can be done with a hostfile. > > >> In any case the point is, using the same program on both lines to me >> seems straightforward enough given that there are numerous instances (as >> ex. 8.13 above) where various switches and options might be different. >> >> I note also that -hosts is different from the -host switch listed as >> pre-defined for mpiexec. >> > > Hydra supports both -host and -hosts. They mean the same thing. > > >> In my case we have a system where the objective is to direct specific >> instances of an application or possibly several to specific hosts - so that >> the mapping can be fixed. In the limit this might consist of -n 1 -host <x> >> <executable> lines, each directing a specific instance of the executable to >> a specific host. >> > > Again, this will not work with today's Hydra. I would suggest playing > around some more with the available options to see if another solution > might work for your use-case. > Back to the real situation. What we (will) have is a machine with a large number of compute nodes. Simply by force of numbers these compute nodes are going to be unreliable: some of them will work, some not, although we do assume that at actual run-time the list of working nodes can be gained by discovery from a root process and configured as a static mapping for the purposes of any given run. The critical point though is that it's only at system initialisation time, not before, that this information can be known. A root process will take a problem (i.e. an application) specification and determine a mapping for the application based on the available hardware topology, the problem topology, and the predicted patterns of use. It then will need to run an MPI_COMM_SPAWN_MULTIPLE to distribute the problem to the working nodes, and we need to make sure the distribution is exactly as specified or the system simply won't work - problem components will be distributed differently from the expected topology, large chunks of data might end up in entirely unexpected places and the application will fall down almost immediately (deadlock is very likely as well). The main thing I need to understand is exactly what the MPI_COMM_SPAWN_MULTIPLE will do. As you can see there are a lot of secondary questions related to things like where I/O goes, what user is on which host, Fortran support etc. but these are only sub-problems of the main issue which is getting to grips with MPI_COMM_SPAWN_MULTIPLE. As you may be able to see, using a hosts file would be awkward at best, because we're not going to know until application startup what the hosts would actuallly be. The best solution I can come up with if it really is the case that this is the only way to map processes to hosts is to have a 2-stage program that does discovery and then pipes the result into a second stage invoked through mpiexec but I'll need to go away and see if there are any interactions with MPI_COMM_SPAWN like what I found related to different users to know if this is going to work. I'll run some tests and let you know what I find. Thanks for your help. > > Ken >

1 0

Re: [mpich-discuss] mpiexec ignoring host setting in configfile
by Kenneth Raffenetti 22 Jun '17

22 Jun '17

On 06/22/2017 11:16 AM, Alexander Rast wrote: > On Thu, Jun 22, 2017 at 4:14 PM, Kenneth Raffenetti > <raffenet(a)mcs.anl.gov <mailto:[email protected]>> wrote: > > -host(s) is a global option. You can only specify it once. > > > ? This doesn't make sense to me. Example 8.13 from the MPI specification > has: > > mpiexec myprog infile1 : myprog infile2 : myprog infile3, corresponding > to Form A > > mpiexec {<above arguments>} : {...} : {...} : ... : { ...} > > which invokes potentially multiple programs with different arguments (of > which host is one) > > and then mentions that Form B: mpiexec -configfile <filename> has lines > of the form separated by the colons in Form A. > > So there's no reason, I would think, that a Form B file can't have > multiple lines with different hosts specified for the same executable. The reason is that Hydra does not support it. You are allowed one -host(s) argument. If you want to submit a patch to allow more than one instance, we will consider it. > > Further, your configfile does not make much sense since your are > using the same program on both lines. You could rather just do: > > mpiexec -n 4 -ppn 2 -hosts Shakespeare,Burns ./mpi_io_test > > > I have not seen the -ppn switch listed anywhere. What does this do? Is > this intended to be a 'limit number of processes to n on the hosts > specified? $ mpiexec -h | grep ppn -ppn processes per node > Notwithstanding this isn't what is needed anyway, because what if I want > specifically to run at most 2 processes on Shakespeare but 5 on Burns > (imagine, for example, that Shakespeare is already loaded with other > processes, so I need to limit it for the executable in concern)? Irregular mappings like this can be done with a hostfile. > > In any case the point is, using the same program on both lines to me > seems straightforward enough given that there are numerous instances (as > ex. 8.13 above) where various switches and options might be different. > > I note also that -hosts is different from the -host switch listed as > pre-defined for mpiexec. Hydra supports both -host and -hosts. They mean the same thing. > > In my case we have a system where the objective is to direct specific > instances of an application or possibly several to specific hosts - so > that the mapping can be fixed. In the limit this might consist of -n 1 > -host <x> <executable> lines, each directing a specific instance of the > executable to a specific host. Again, this will not work with today's Hydra. I would suggest playing around some more with the available options to see if another solution might work for your use-case. Ken _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec ignoring host setting in configfile
by Alexander Rast 22 Jun '17

22 Jun '17

On Thu, Jun 22, 2017 at 4:14 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > -host(s) is a global option. You can only specify it once. > ? This doesn't make sense to me. Example 8.13 from the MPI specification has: mpiexec myprog infile1 : myprog infile2 : myprog infile3, corresponding to Form A mpiexec {<above arguments>} : {...} : {...} : ... : { ...} which invokes potentially multiple programs with different arguments (of which host is one) and then mentions that Form B: mpiexec -configfile <filename> has lines of the form separated by the colons in Form A. So there's no reason, I would think, that a Form B file can't have multiple lines with different hosts specified for the same executable. > Further, your configfile does not make much sense since your are using the > same program on both lines. You could rather just do: > > mpiexec -n 4 -ppn 2 -hosts Shakespeare,Burns ./mpi_io_test > I have not seen the -ppn switch listed anywhere. What does this do? Is this intended to be a 'limit number of processes to n on the hosts specified? Notwithstanding this isn't what is needed anyway, because what if I want specifically to run at most 2 processes on Shakespeare but 5 on Burns (imagine, for example, that Shakespeare is already loaded with other processes, so I need to limit it for the executable in concern)? In any case the point is, using the same program on both lines to me seems straightforward enough given that there are numerous instances (as ex. 8.13 above) where various switches and options might be different. I note also that -hosts is different from the -host switch listed as pre-defined for mpiexec. In my case we have a system where the objective is to direct specific instances of an application or possibly several to specific hosts - so that the mapping can be fixed. In the limit this might consist of -n 1 -host <x> <executable> lines, each directing a specific instance of the executable to a specific host. > > Ken > > On 06/22/2017 09:55 AM, Alexander Rast wrote: > >> I'm getting a very suprising result with the attached application and >> configfile. As you can see, I'm asking MPI to run 2 processes on >> Shakespeare, and 2 on Burns. But when I ran the program using mpiexec >> -configfile ./MPI_config.txt what I actually get is the output shown in >> MPI_IO_Test.txt. Furthermore I can confirm that this is genuine: I shut >> down Burns altogether and the same result ensues. It's as if mpiexec is >> taking the -host field as advisory but not required, that is, it might >> execute on the host asked for, but then again if it decides not to, it >> doesn't interpret the -host field as a requirement. I get the distinct >> impression, indeed, that instead of interpreting the host as a requirement, >> it uses a greedy process allocation: if it can spawn all of the processes >> on the invoking host (the one that ran mpiexec) it does so, and only once >> the resources there are exhausted does it look at other possible hosts. >> >> Clearly if you specify a -host the intention is that the process should >> be placed there. This looks like a bug in mpiexec. >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >>

1 0

Re: [mpich-discuss] mpiexec ignoring host setting in configfile
by Kenneth Raffenetti 22 Jun '17

22 Jun '17

-host(s) is a global option. You can only specify it once. Further, your configfile does not make much sense since your are using the same program on both lines. You could rather just do: mpiexec -n 4 -ppn 2 -hosts Shakespeare,Burns ./mpi_io_test Ken On 06/22/2017 09:55 AM, Alexander Rast wrote: > I'm getting a very suprising result with the attached application and > configfile. As you can see, I'm asking MPI to run 2 processes on > Shakespeare, and 2 on Burns. But when I ran the program using mpiexec > -configfile ./MPI_config.txt what I actually get is the output shown in > MPI_IO_Test.txt. Furthermore I can confirm that this is genuine: I shut > down Burns altogether and the same result ensues. It's as if mpiexec is > taking the -host field as advisory but not required, that is, it might > execute on the host asked for, but then again if it decides not to, it > doesn't interpret the -host field as a requirement. I get the distinct > impression, indeed, that instead of interpreting the host as a > requirement, it uses a greedy process allocation: if it can spawn all of > the processes on the invoking host (the one that ran mpiexec) it does > so, and only once the resources there are exhausted does it look at > other possible hosts. > > Clearly if you specify a -host the intention is that the process should > be placed there. This looks like a bug in mpiexec. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpiexec ignoring host setting in configfile
by Alexander Rast 22 Jun '17

22 Jun '17

I'm getting a very suprising result with the attached application and configfile. As you can see, I'm asking MPI to run 2 processes on Shakespeare, and 2 on Burns. But when I ran the program using mpiexec -configfile ./MPI_config.txt what I actually get is the output shown in MPI_IO_Test.txt. Furthermore I can confirm that this is genuine: I shut down Burns altogether and the same result ensues. It's as if mpiexec is taking the -host field as advisory but not required, that is, it might execute on the host asked for, but then again if it decides not to, it doesn't interpret the -host field as a requirement. I get the distinct impression, indeed, that instead of interpreting the host as a requirement, it uses a greedy process allocation: if it can spawn all of the processes on the invoking host (the one that ran mpiexec) it does so, and only once the resources there are exhausted does it look at other possible hosts. Clearly if you specify a -host the intention is that the process should be placed there. This looks like a bug in mpiexec. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Extent of support for MPI_T_... interface?
by Halim Amer 19 Jun '17

19 Jun '17

The type of variables you can measure should all be visible at init time. For the same type of variable, however, you can have multiple bindings to different MPI objects. For example, when you create a new communicator, you can bind a new performance variable to that communicator instead of tracking the same performance metric globally on all communicators. You can refer to section 14.3.2 of the MPI 3.1 standard for more details. Halim www.mcs.anl.gov/~aamer On 6/19/17 11:07 AM, Alexander Rast wrote: > OK that's working now. I see 10 performance variables. Are any more likely > to become available as processes/comms are started? In particular what will > happen if I run MPI_Comm_spawn to generate new processes? > > Thanks for the help thus far. > > > > On Mon, Jun 19, 2017 at 3:40 PM, Halim Amer <aamer(a)anl.gov> wrote: > >> Tracking performance variables incurs overhead, thus disabled by default. >> You can enable tracking pvars with --enable-mpit-pvars at configure time. >> Feel free to look at "configure --help" to get what pvars are tracked in >> the MPICH version you are using. >> >> Halim >> www.mcs.anl.gov/~aamer >> >> >> On 6/19/17 9:27 AM, Alexander Rast wrote: >> >>> I just ran a simple program to output the list of performance variables >>> available at MPI initialisation. That it compiled and ran indicates that >>> MPICH does at least support the interface, however it returned 0 variables >>> available. Is this true - that MPICH supports the interface but exports no >>> performance variables? >>> >>> The other possible scenario I could envisage is that MPICH exports >>> possibly >>> many variables but these aren't available at initialisation time, rather >>> they are dynamically exported as processes and communicators are created. >>> However, if this is the case I don't understand the expected usage model. >>> It's not clear how you could make use of this data without very awkward >>> processing because the underlying datatypes are only exported as an >>> MPI_Datatype opaque handle. >>> >>> Does either of these 2 scenarios reflect the extent of MPI_T support? >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Extent of support for MPI_T_... interface?
by Halim Amer 19 Jun '17

19 Jun '17

> I'd read through configure --help multiple times and not noticed > this option You were interested in MPI_T performance variables, which is clearly indicated in that configure option. I don't understand how you missed that. > which requires a very close read to interpret the implications. It's not > the kind of thing you're going to spot immediately even ex post facto > without significant prior knowledge. It's not documented or mentioned > anywhere else. Sure, the documentation needs some improvements, but as you guessed, our plates are full of with higher priority items. If you have concrete suggestions, feel free to let us know or contribute patches. Keep in mind, however, that what MPI_T exposes is MPI implementation dependent. That means, since we are talking about MPICH, you might need to be familiar with the MPICH software stack to understand what you are measuring. The easiest way for you to understand what performance variables we expose, is to pass "all" to --enable-mpit-pvars and print the description of each performance variable (using MPI_T_pvar_get_info). While some variables are self explanatory and could be applicable to many MPI implementations (e.g., length of message queues), others, such as those related to the Nemesis software layer, are MPICH-specific and will require understand MPICH deeper. > I also think this would more appropriately be a run-time option for mpiexec > rather than a build-time option for the toolchain itself. First, multiple builds of the same MPI implementation with different performance variables is allowed by the standard. From the MPI 3.1 Standard: "The number and type of control variables and performance variables can vary between MPI implementations, platforms and different builds of the same implementation on the same platform as well as between runs. Hence, any application relying on a particular variable will not be portable. Further, there is no guarantee that the number of variables and variable indices are the same across connected processes." If you care about portability, then don't use the MPI_T interface. Second, having runtime options to enable-disable tracking performance variables would add non-negligible overhead that most users would complain about. For instance, adding branches on the critical path (e.g., if(message_queue_tracking_is_active) on a MPI_Isend or MPI_Put path) to support such runtime option would be too expensive for fine-grained communication. Users want MPI communication calls to get translated to native hardware with the lowest overhead possible. > By the way, just looking through the configure --help output again I notice > that at least one option: --enable-error-messages-level, doesn't document > what the default setting actually is. There may be others. Thanks for reporting this, we will fix them. Halim www.mcs.anl.gov/~aamer On 6/19/17 10:27 AM, Alexander Rast wrote: > I have to comment that clearly the documentation needs improvement. It's > always the last on the priority list, I realise, but this is really very > obscure. I'd read through configure --help multiple times and not noticed > this option, and even now the description is highly non-obvious if you > didn't already know what you were looking for and what it meant. The line > reads: > > '--enable-mpit-pvars=list - selectively enable MPI_T performance variable > in > modules. list is a comma-separated module > names, > including (Default is "none"): > none - No performance info recorded > recvq - All message queue-related > nem - All nemesis-related > rma - All rma-related > all - All variables above' > > which requires a very close read to interpret the implications. It's not > the kind of thing you're going to spot immediately even ex post facto > without significant prior knowledge. It's not documented or mentioned > anywhere else. > > I also think this would more appropriately be a run-time option for mpiexec > rather than a build-time option for the toolchain itself. There are > numerous situations where I'd want to have performance monitoring enabled > for some runs but not others. It doesn't seem to me to be an obvious choice > to make it a build-time option, and it would be awkward to have 2 separate > builds. In addition there are the MPICH binary installs available for many > OS's which ought to have these options available in some form - it would be > a surprise I suspect to a user who'd installed from a binary image to find > support disabled for certain features that they'd not even been given the > option to select. > > By the way, just looking through the configure --help output again I notice > that at least one option: --enable-error-messages-level, doesn't document > what the default setting actually is. There may be others. > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0