March 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Hydra fills in the MPIR_proctable[] incorrectly with multiple processes per node
by John DelSignore 18 Mar '13

18 Mar '13

Pavan Balaji wrote: > Hi John, > > On 03/12/2013 11:32 AM US Central Time, John DelSignore wrote: >> If possible, it would be good is MPICH could be changed to share the >> host and executable name character strings across multiple process >> descriptor entries. > > This is a maintenance hassle with respect to chasing dangling pointers > in the future. I've never looked at the MPICH implementation, so I don't understand the issue with dangling pointers. However, I do know that other MPI implementations support common strings for the MPIR proctable and as a result we have seen significant performance benefits when extracting the MPIR proctable. > Also, I'm not sure how much space this saves. It's not purely a matter of saving space (which can be significant), it's also a matter of saving time when the tool attempts to extract the strings from the starter process. On systems like IBM Blue Gene/Q we are beginning to deal with MPI jobs approaching one million processes or more. Let's do a rough calculation on a BG/Q job that has 1,048,576 processes spread across 65,536 compute nodes (16 processes per node), with an IO-node to compute-node ratio of 128:1. Also assume that the absolute path to the executable is "only" 50 characters long, it's an SPMD code (one executable for all processes), and the that host name strings are IP addresses that average 15 characters long. Without the "common string" optimization, the executable strings would occupy at least 50MB of space, and the host name strings would occupy 15MB of space. But the thing that really hurts is that the tool would have to read 2 million separate strings using an interface like ptrace() or /proc, which can be slow. With the "common string" optimization, the executable string would occupy about 50 bytes of space, and the host name strings would occupy 7,680 bytes ((1,048,576 / 16 / 128) * 15). But the bigger win is that tool would have to read only 513 separate strings using ptrace() or /proc, which represents a big time savings. > We are working with the MPI Forum on a more scalable interface where > this information is distributed across various processes. That might be > a better model here. I'm on the MPI Tool Working group, and we are many many miles away from a new MPIR interface, which we've been calling "MPIR v2". There is no MPIR v2 proposal on the table, and there seems to be virtually no interest in creating one. And from my perspective, IBM has proven that MPIR v1 can scale to millions of processes, so it seems unlikely to me that there will be an MPIR v2 anytime soon. Besides, I think the real motivator for an MPIR v2 will be to support MPI dynamic process creation, not scalability, but there doesn't seem to be much demand for that either. Cheers, John D. > -- Pavan >

1 0

Re: [mpich-discuss] Hi Jayesh
by Jayesh Krishna 18 Mar '13

18 Mar '13

Hi, MPICH allows compiling+running MPI programs on a cluster of computers (or multi-core machine[s]). So to use MPICH the software that you run on your machine has to be rewritten using MPI and built using MPICH. Let us know if you need further information. Regards, Jayesh ----- Original Message ----- From: "Javad Ataiy" <javadataiy(a)gmail.com> To: mpich-discuss(a)mcs.anl.gov Sent: Sunday, March 17, 2013 1:00:17 AM Subject: [mpich-discuss] Hi Jayesh I have an AMD Athlon PC x86 Desktop, and ThinkPad x220 Core i7 laptop. I want to increase my desktop processing power, can this be done with MPICH. I use desktop mostly for gaming. Thanks alot. Sincerely JAVAD ATAIY _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] discuss Digest, Vol 5, Issue 4
by Muhammad Zulfikar Handana 18 Mar '13

18 Mar '13

hello, i have some questions about MPI-I/O. 1) when i trying execute my program using MPI-I/O. why when iam execute with 1 proses, execute time more good than when iam execute with 2 proses. am i wrong? this is my source code. MPI_File_open (MPI_COMM_WORLD, file_akhir, MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &hasil); int jumlah1 = (((block-1)*16)+pad); int jumlah = jumlah1 / size; MPI_File_set_view(hasil, rank * jumlah * sizeof(char),MPI_CHAR, MPI_CHAR, "native", MPI_INFO_NULL); MPI_File_write (hasil, output, jumlah1, MPI_CHAR, MPI_STATUS_IGNORE); MPI_File_close(&hasil); 2) and after that i have compared that source using MPI - I/O and with sequential I/O, apparently, sequential i / o is better than the parallel i / o. how it could be happen? this is my sequential source. if (rank==0) { hasil = fopen (file_akhir, "wb"); int test = (((block-1)*16)+pad); fwrite (output, sizeof(char), test, hasil ); fclose (hasil); } thanks a lot for your attention

1 0

Hi Jayesh
by Javad Ataiy 17 Mar '13

17 Mar '13

I have an AMD Athlon PC x86 Desktop, and ThinkPad x220 Core i7 laptop. I want to increase my desktop processing power, can this be done with MPICH. I use desktop mostly for gaming. Thanks alot. Sincerely JAVAD ATAIY

1 0

Re: [mpich-discuss] Hydra fills in the MPIR_proctable[] incorrectly with multiple processes per node
by Pavan Balaji 15 Mar '13

15 Mar '13

Hi John, On 03/12/2013 11:32 AM US Central Time, John DelSignore wrote: > I look forward to your fix for this problem. This is fixed in the git master. http://git.mpich.org/mpich.git/commitdiff/e04dd4b64ff618f2df58789265b741a8e… It should appear in the upcoming 3.0.3 release. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Hydra fills in the MPIR_proctable[] incorrectly with multiple processes per node
by Pavan Balaji 14 Mar '13

14 Mar '13

Hi John, On 03/12/2013 11:32 AM US Central Time, John DelSignore wrote: > If possible, it would be good is MPICH could be changed to share the > host and executable name character strings across multiple process > descriptor entries. This is a maintenance hassle with respect to chasing dangling pointers in the future. Also, I'm not sure how much space this saves. We are working with the MPI Forum on a more scalable interface where this information is distributed across various processes. That might be a better model here. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] MPI_set_view, for offset larger than an integer
by Rajeev Thakur 13 Mar '13

13 Mar '13

And if you are using Fortran, you have to define the offset variable as integer (kind=MPI_OFFSET_KIND). On Mar 13, 2013, at 10:57 AM, Jeff Hammond wrote: > The MPI standard, which is the only official documentation about the > function syntax of MPI_File_set_view, says this: > > int MPI_File_set_view(MPI_File fh, MPI_Offset disp, MPI_Datatype etype, > MPI_Datatype filetype, const char *datarep, MPI_Info info) > > The size of the disp type is specified as follows: > > 13.6.7 MPI_Offset Type > > MPI_Offset is an integer type of size sufficient to represent the size > (in bytes) of the largest file supported by MPI. Displacements and > offsets are always specified as values of type MPI_Offset. > > Thus, it should not be possible to overflow this value if you're > accessing the file in a correct way. > > Jeff > > On Wed, Mar 13, 2013 at 10:28 AM, Ana María Sánchez > <krazorange(a)gmail.com> wrote: >> Hi all, >> >> I need to read a file, using MPI_set_view. The problem occurs when the >> file is so large that the offset that I need to provide to the MPI_set_view >> is larger than the maximum value for an integer. For example I need to set >> the offset to 3826145886. In some documentation I've read the offset is a >> long, but in other it indicates its an integer. >> >> Is there any possible work around this? Could the offset be set to >> negative, and if so, will it start at the end of the file? >> >> I have installed mpich2-1.3.1. >> >> Thank you, >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_set_view, for offset larger than an integer
by Jeff Hammond 13 Mar '13

13 Mar '13

The MPI standard, which is the only official documentation about the function syntax of MPI_File_set_view, says this: int MPI_File_set_view(MPI_File fh, MPI_Offset disp, MPI_Datatype etype, MPI_Datatype filetype, const char *datarep, MPI_Info info) The size of the disp type is specified as follows: 13.6.7 MPI_Offset Type MPI_Offset is an integer type of size sufficient to represent the size (in bytes) of the largest file supported by MPI. Displacements and offsets are always specified as values of type MPI_Offset. Thus, it should not be possible to overflow this value if you're accessing the file in a correct way. Jeff On Wed, Mar 13, 2013 at 10:28 AM, Ana María Sánchez <krazorange(a)gmail.com> wrote: > Hi all, > > I need to read a file, using MPI_set_view. The problem occurs when the > file is so large that the offset that I need to provide to the MPI_set_view > is larger than the maximum value for an integer. For example I need to set > the offset to 3826145886. In some documentation I've read the offset is a > long, but in other it indicates its an integer. > > Is there any possible work around this? Could the offset be set to > negative, and if so, will it start at the end of the file? > > I have installed mpich2-1.3.1. > > Thank you, > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

MPI_set_view, for offset larger than an integer
by Ana María Sánchez 13 Mar '13

13 Mar '13

Hi all, I need to read a file, using MPI_set_view. The problem occurs when the file is so large that the offset that I need to provide to the MPI_set_view is larger than the maximum value for an integer. For example I need to set the offset to 3826145886. In some documentation I've read the offset is a long, but in other it indicates its an integer. Is there any possible work around this? Could the offset be set to negative, and if so, will it start at the end of the file? I have installed mpich2-1.3.1. Thank you,

1 0

Re: [mpich-discuss] Possible bug: --enable-g=all configure option does not include all other --enable-g options
by Dave Goodell 12 Mar '13

12 Mar '13

fixed: http://git.mpich.org/mpich.git/commitdiff/4b6df5cd -Dave On Mar 12, 2013, at 10:42 AM CDT, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > I don't like the gcc model that much. > > The key is just to change the current assumption that "yes"=="all". I'll code up a patch to add a "most" and make "yes" (i.e., the default when you just pass "--enable-g") equivalent to that. Then I'll change "all" to really mean everything. > > -Dave > > On Mar 12, 2013, at 10:36 AM CDT, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > >> How about, following the gcc model: >> >> --enable-g=all enable most debugging features >> --enable-g=extra enable most + additional debugging features >> >> ~Jim. >> >> On 3/12/13 8:59 AM, William Gropp wrote: >>> While I agree that "all" should mean "all", this unfortunate use is >>> consistent with gcc, which uses -Wall to mean "most of the warning >>> options, but not the ones that who ever is in charge of this release did >>> not like or forgot about." >>> >>> A better name might be --enable-g=most - and for gcc, -Wmost . Anything >>> else is either too long or misleading. >>> >>> Bill >>> >>> William Gropp >>> Director, Parallel Computing Institute >>> Deputy Director for Research >>> Institute for Advanced Computing Applications and Technologies >>> Thomas M. Siebel Chair in Computer Science >>> University of Illinois Urbana-Champaign >>> >>> >>> >>> >>> On Mar 11, 2013, at 8:14 PM, Jeff Hammond wrote: >>> >>>> --enable-g=all should enable all debugging options to satisfy the >>>> principle of least surprise as well as simple English language meaning >>>> of the word "all". >>>> >>>> perhaps there should be an option >>>> --enable-g=important/most/devel/useful/sane that enables all of the >>>> debug options that are reasonable for the "average hardcore user" / >>>> developer. >>>> >>>> jeff >>>> >>>> On Mon, Mar 11, 2013 at 4:33 PM, Jeremiah Willcock >>>> <jewillco(a)osl.iu.edu <mailto:[email protected]>> wrote: >>>>> On Mon, 11 Mar 2013, Dave Goodell wrote: >>>>> >>>>>> On Mar 11, 2013, at 3:36 PM CDT, Jeremiah Willcock >>>>>> <jewillco(a)osl.iu.edu <mailto:[email protected]>> >>>>>> wrote: >>>>>> >>>>>>> In MPICH 3.0.2, the help message for the --enable-g=all configure >>>>>>> option >>>>>>> states that it enables "All of the above choices", but in fact >>>>>>> (according to >>>>>>> configure.ac) it does not enable the "handle" or "memarena" >>>>>>> debugging flags. >>>>>>> Is that the intended behavior? Thank you for your help. >>>>>> >>>>>> >>>>>> The situation is unclear. I just did a little bit of git >>>>>> archaeology and >>>>>> it looks like these flags were added prior to MPICH2-1.0.6, which is >>>>>> as far >>>>>> back as I can look without having to sift through ancient CVS >>>>>> history. The >>>>>> pickaxe doesn't show any place where variables corresponding to >>>>>> those flags >>>>>> might have been removed from the "all|yes)" case. It looks like just a >>>>>> mistake or sloppiness on someone's part in the distant past. >>>>>> >>>>>> I'm not sure what the right "fix" is. The "memarena" option actually >>>>>> introduces quite a bit of overhead that we may not want enabled >>>>>> casually. >>>>>> The "handles" option is pretty lightweight, at least in stock MPICH. I >>>>>> suppose we could add both to the "all" option and then change the "yes" >>>>>> option to be a set of reasonable defaults instead. >>>>>> >>>>>> Do you have a preference? Or are you just observing a discrepancy? >>>>> >>>>> >>>>> I was both observing a discrepancy and trying to debug something and was >>>>> surprised that "all" didn't include "handles". >>>>> >>>>> -- Jeremiah Willcock >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> Argonne Leadership Computing Facility >>>> University of Chicago Computation Institute >>>> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 >>>> http://www.linkedin.com/in/jeffhammond >>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0