April 2014 - devel - mailman3.cels.anl.gov

Issue with MPI_Error_string() for user-defined codes/clases
by Lisandro Dalcin 22 Apr '14

22 Apr '14

The MPI-3 standard says (pp.354, lines 39-40): """ If MPI_ERROR_STRING is called when no string has been set, it will return a empty string (all spaces in Fortran, "" in C). """ The following simple tests segfaults. A quick fix would be to return an empty string in MPIR_Err_get_dynerr_string() (file dynerrutil.c), e.g: diff --git a/src/mpi/errhan/dynerrutil.c b/src/mpi/errhan/dynerrutil.c index 943e8c3..fb40469 100644 --- a/src/mpi/errhan/dynerrutil.c +++ b/src/mpi/errhan/dynerrutil.c @@ -297,11 +297,13 @@ const char *MPIR_Err_get_dynerr_string( int code ) if (errcode) { if (errcode < first_free_code) { errstr = user_code_msgs[errcode]; + if (!errstr) errstr = ""; } } else { if (errclass < first_free_class) { errstr = user_class_msgs[errclass]; + if (!errstr) errstr = ""; } } [dalcinl@kw2060 ~]$ cat error_string2.c #include <stdio.h> #include <mpi.h> int main(int argc, char *argv[]) { int errorclass; char errorstring[MPI_MAX_ERROR_STRING] = {64,0}; int slen; MPI_Init(&argc, &argv); MPI_Add_error_class(&errorclass); MPI_Error_string(errorclass, errorstring, &slen); printf("errorclass:%d errorstring:'%s' len:%d\n", errorclass, errorstring, slen); MPI_Finalize(); return 0; } [dalcinl@kw2060 ~]$ mpicc error_string2.c [dalcinl@kw2060 ~]$ ./a.out Segmentation fault (core dumped) [dalcinl@kw2060 ~]$ valgrind -q ./a.out ==14176== Invalid read of size 1 ==14176== at 0x4C6F5E7: MPIU_Strncpy (safestr.c:65) ==14176== by 0x4C61864: MPIR_Err_get_string (errutil.c:601) ==14176== by 0x4DA4DB4: PMPI_Error_string (error_string.c:80) ==14176== by 0x400888: main (in /home/dalcinl/Devel/BUGS-MPI/~/a.out) ==14176== Address 0x0 is not stack'd, malloc'd or (recently) free'd ==14176== ==14176== ==14176== Process terminating with default action of signal 11 (SIGSEGV) ==14176== Access not within mapped region at address 0x0 ==14176== at 0x4C6F5E7: MPIU_Strncpy (safestr.c:65) ==14176== by 0x4C61864: MPIR_Err_get_string (errutil.c:601) ==14176== by 0x4DA4DB4: PMPI_Error_string (error_string.c:80) ==14176== by 0x400888: main (in /home/dalcinl/Devel/BUGS-MPI/~/a.out) ==14176== If you believe this happened as a result of a stack ==14176== overflow in your program's main thread (unlikely but ==14176== possible), you can try to increase the size of the ==14176== main thread stack using the --main-stacksize= flag. ==14176== The main thread stack size used in this run was 8720384. Segmentation fault (core dumped) -- Lisandro Dalcin --------------- CIMEC (UNL/CONICET) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1016) Tel/Fax: +54-342-4511169

1 0

Re: [mpich-devel] Should MPI_{Comm|Win|File}_Set_info(..., MPI_INFO_NULL) fail ?
by Rajeev Thakur 09 Apr '14

09 Apr '14

It is not clear that passing MPI_INFO_NULL should result in resetting hints to default. For example, pg 248 at the bottom says "When an info object that specifies a subset of valid hints is passed to MPI_COMM_SET_INFO, there will be no effect on previously set or defaulted hints that the info does not specify." So if MPI_INFO_NULL is passed, at best nothing should happen. But the function definition doesn't say that INFO_NULL is a valid parameter. The behavior you are seeing is because comm_set_info uses a different (probably buggy) null test MPIR_ERRTEST_ARGNULL(info, "info", mpi_errno); whereas win_set_info uses MPIR_ERRTEST_INFO(info, mpi_errno); and I/O uses something else :-). Rajeev On Apr 9, 2014, at 4:51 AM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > Right now, it success for Comm but fails for Win and File. IMHO, all > should success, this should be a way for users to ask the > implementation "please, reset the info hints to whatever the defaults > are". > > If you decide they should fail, then the Comm case needs a fix to > generate an error. > > $ mpicc comm_set_info.c > $ ./a.out > > > $ mpicc win_set_info.c > $ ./a.out > Fatal error in MPI_Win_set_info: Invalid argument, error stack: > MPI_Win_set_info(126): MPI_Win_set_info(win=win=0xa0000000, > info=MPI_INFO_NULL) failed > MPI_Win_set_info(82).: Null MPI_Info > > > $ mpicc file_set_info.c > $ ./a.out > Fatal error in PMPI_Info_dup: Invalid argument, error stack: > PMPI_Info_dup(149): MPI_Info_dup(MPI_INFO_NULL, newinfo=0x7fff58b899c4) failed > PMPI_Info_dup(108): Null MPI_Info > > > PS: Perhaps you have an easy fix for the error message (PMPI -> MPI) > in the last one? > > -- > Lisandro Dalcin > --------------- > CIMEC (UNL/CONICET) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1016) > Tel/Fax: +54-342-4511169 > <comm_set_info.c><file_set_info.c><win_set_info.c>

1 0

Should MPI_{Comm|Win|File}_Set_info(..., MPI_INFO_NULL) fail ?
by Lisandro Dalcin 09 Apr '14

09 Apr '14

Right now, it success for Comm but fails for Win and File. IMHO, all should success, this should be a way for users to ask the implementation "please, reset the info hints to whatever the defaults are". If you decide they should fail, then the Comm case needs a fix to generate an error. $ mpicc comm_set_info.c $ ./a.out $ mpicc win_set_info.c $ ./a.out Fatal error in MPI_Win_set_info: Invalid argument, error stack: MPI_Win_set_info(126): MPI_Win_set_info(win=win=0xa0000000, info=MPI_INFO_NULL) failed MPI_Win_set_info(82).: Null MPI_Info $ mpicc file_set_info.c $ ./a.out Fatal error in PMPI_Info_dup: Invalid argument, error stack: PMPI_Info_dup(149): MPI_Info_dup(MPI_INFO_NULL, newinfo=0x7fff58b899c4) failed PMPI_Info_dup(108): Null MPI_Info PS: Perhaps you have an easy fix for the error message (PMPI -> MPI) in the last one? -- Lisandro Dalcin --------------- CIMEC (UNL/CONICET) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1016) Tel/Fax: +54-342-4511169

1 0

Default print name for MPI-3 C++ datatypes
by Lisandro Dalcin 09 Apr '14

09 Apr '14

Please review the following trivial patch (also attached), diff --git a/src/mpi/datatype/type_get_name.c b/src/mpi/datatype/type_get_name.c index 07c2f84..e550dfb 100644 --- a/src/mpi/datatype/type_get_name.c +++ b/src/mpi/datatype/type_get_name.c @@ -108,6 +108,12 @@ static mpi_names_t mpi_names[] = { type_name_entry(MPI_C_DOUBLE_COMPLEX), type_name_entry(MPI_C_LONG_DOUBLE_COMPLEX), + /* C++ types */ + type_name_entry(MPI_CXX_BOOL), + type_name_entry(MPI_CXX_FLOAT_COMPLEX), + type_name_entry(MPI_CXX_DOUBLE_COMPLEX), + type_name_entry(MPI_CXX_LONG_DOUBLE_COMPLEX), + /* address/offset types */ type_name_entry(MPI_AINT), type_name_entry(MPI_OFFSET), -- Lisandro Dalcin --------------- CIMEC (UNL/CONICET) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1016) Tel/Fax: +54-342-4511169

1 0

Re: [mpich-devel] romio questions
by Balaji, Pavan 04 Apr '14

04 Apr '14

Every time the MPICH stack is “stuck” in the progress engine (i.e., nothing happened for some time), it’ll release the lock so other threads can grab it. So unless File_write_all is blocking on a file-system operation, other threads should be able to make progress just fine. — Pavan On Apr 4, 2014, at 4:34 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > This is getting out of my depth. MPICH guys: Mark Allen is updating Platform's MPI and has been asking me some questions about multi-threaded ROMIO. > > The independent ROMIO routines might make some local MPI calls to process the data type, but otherwise, yes, the only blocking call would be the file system operation. > > I haven't thought about what would happen if MPI_File_write_all was called from two threads. > > MPICH guys: there's no way the CS_ENTER/EXIT macros can be that clever, right? > > ==rob > > On 04/04/2014 04:28 PM, Mark Allen wrote: >> Thanks. I looked a bit at the CS_ENTER/EXIT code. Am I right that all >> the non-collective MPIO calls like MPI_File_read etc are only blocking >> in the sense of waiting on some local operation to complete? If that's >> the case it would be okay for them to hold a lock from beginning to end >> of the call like that. But for the collective MPIO operations I don't >> think you could just hold the lock for the entire function call since >> you might end up waiting for remote peers to arrive, and if two >> collectives were taking place at the same time they could mismatch and >> deadlock. But I'm not sure if that's what the code is doing or not. Is >> there a release of the lock hidden somewhere when collectives block? If >> inside the MPI_Allgather for example MPI performed a release and regain >> of the MPIO lock is the blocking MPIO collective holding any shared >> resource that would get corrupted by letting other threads in at that point? >> >> Fwiw I also noticed mpi-io/ioreq_f2c.c looks to be a case where an early >> return runs the risk of a mismatched CS_ENTER w/o a corresponding >> CS_EXIT. It looks like all the other files used goto fn_exit to ensure >> a match. > > >> >> Mark >> >> >> Inactive hide details for Rob Latham ---04/04/2014 09:49:00 AM---On >> 04/04/2014 01:22 AM, Paul Coffman wrote: > ----- Forwarded Rob Latham >> ---04/04/2014 09:49:00 AM---On 04/04/2014 01:22 AM, Paul Coffman wrote: >> > ----- Forwarded by Paul Coffman/Rochester/IBM on 04/04 >> >> From: Rob Latham <robl(a)mcs.anl.gov> >> To: Paul Coffman/Rochester/IBM@IBMUS >> Cc: Mark Allen/Dallas/IBM@IBMUS >> Date: 04/04/2014 09:49 AM >> Subject: Re: Fw: romio questions >> >> ------------------------------------------------------------------------ >> >> >> >> >> >> On 04/04/2014 01:22 AM, Paul Coffman wrote: >> >> > ----- Forwarded by Paul Coffman/Rochester/IBM on 04/04/2014 01:22 AM >> ----- >> > >> > From: Mark Allen/Dallas/IBM >> > To: Paul Coffman/Rochester/IBM@IBMUS, >> > Date: 04/04/2014 01:07 AM >> > Subject: romio questions >> > ------------------------------------------------------------------------ >> > >> > >> > I have two questions/topics for you: >> > >> > First I wanted to ask do you happen to know if romio is thread safe? I >> > see a fair number of critical-section begin/end macros and am guessing >> > it is, but thought I'd ask anyway. >> >> the internal romio routines (things in the ADIO and ADIOI namespace) >> rely on several bits of global state -- the flattened representation of >> the file and memory datatype come to mind first, but there are probably >> others. The critical section macros are at the MPI-IO interface to >> romio and should provide a "big lock" to the rest. >> >> I haven't tried this though. I would be a little nervous about it >> working without a few patches. >> >> > Second, I noticed romio uses an extension of generalized requests >> > described here: >> > >> http://www.cs.illinois.edu/~wgropp/bib/papers/2007/latham_grequest-enhance-… >> > >> > but the code looks confused on whether the proposed wait_fn callback is >> > waiting for a single request or all the requests. >> > >> > In romio's definitions of >> > ADIOI_PVFS2_aio_wait_fn >> > ADIOI_GEN_aio_wait_fn >> > the wait_fn looks to me very much like it's waiting for all the reqs, vs >> > the NTFS function that looks like it's waiting on some >> > ADIOI_NTFS_aio_wait_fn >> >> the intent for the extended wait_fn was indeed to wait for all >> outstanding generalized requests. Specifically, to call aio_suspend on >> more than one operation at a time. >> >> I would not read too much into NTFS_aio_wait_fn. that work was done >> quickly for the paper and then set aside. >> >> > And where these wait_fn callbacks get used, MPI_Waitsome uses >> > MPIR_Grequest_progress_poke which conditionally calls wait_fn. It seems >> > to me this would make MPI_Waitsome erroneously block until all its >> > generalized requests finish (the code in MPIR_Grequest_progress_poke >> > looks like it believes the wait_fn is supposed to just complete one >> > request). >> >> It has been some years since I looked at the waitsome/extended-grequest >> interaction, but it does sound like I could have implemented it better... >> >> > Is this an area you've worked with any or something we need to worry >> > about? The reason I was looking at it was in order to pull in the new >> > romio I figured I'd just add MPICH's concept of extended generalized >> > requests into our MPI, so I wanted to make sure I understood how they >> > were intended to work in mpich and I think it's pretty muddled there >> > unless I'm reading it wrong. >> >> You would be the first developers besides me to look closely at our >> extended generalized request proposal. It's no surprise to me it's >> muddled, since it hasn't had a lot of attention over the years. >> >> at various points in the last 6 years MPICH developers have modified the >> implementation of generalized requests to keep the common code paths >> speedy. That might explain some murkiness or things like 'waitsome' >> actually waiting for everything. >> >> ==rob >> >> -- >> Rob Latham >> Mathematics and Computer Science Division >> Argonne National Lab, IL USA >> >> > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA

1 0

Re: [mpich-devel] Fw: romio questions
by Rob Latham 04 Apr '14

04 Apr '14

This is getting out of my depth. MPICH guys: Mark Allen is updating Platform's MPI and has been asking me some questions about multi-threaded ROMIO. The independent ROMIO routines might make some local MPI calls to process the data type, but otherwise, yes, the only blocking call would be the file system operation. I haven't thought about what would happen if MPI_File_write_all was called from two threads. MPICH guys: there's no way the CS_ENTER/EXIT macros can be that clever, right? ==rob On 04/04/2014 04:28 PM, Mark Allen wrote: > Thanks. I looked a bit at the CS_ENTER/EXIT code. Am I right that all > the non-collective MPIO calls like MPI_File_read etc are only blocking > in the sense of waiting on some local operation to complete? If that's > the case it would be okay for them to hold a lock from beginning to end > of the call like that. But for the collective MPIO operations I don't > think you could just hold the lock for the entire function call since > you might end up waiting for remote peers to arrive, and if two > collectives were taking place at the same time they could mismatch and > deadlock. But I'm not sure if that's what the code is doing or not. Is > there a release of the lock hidden somewhere when collectives block? If > inside the MPI_Allgather for example MPI performed a release and regain > of the MPIO lock is the blocking MPIO collective holding any shared > resource that would get corrupted by letting other threads in at that point? > > Fwiw I also noticed mpi-io/ioreq_f2c.c looks to be a case where an early > return runs the risk of a mismatched CS_ENTER w/o a corresponding > CS_EXIT. It looks like all the other files used goto fn_exit to ensure > a match. > > Mark > > > Inactive hide details for Rob Latham ---04/04/2014 09:49:00 AM---On > 04/04/2014 01:22 AM, Paul Coffman wrote: > ----- Forwarded Rob Latham > ---04/04/2014 09:49:00 AM---On 04/04/2014 01:22 AM, Paul Coffman wrote: > > ----- Forwarded by Paul Coffman/Rochester/IBM on 04/04 > > From: Rob Latham <robl(a)mcs.anl.gov> > To: Paul Coffman/Rochester/IBM@IBMUS > Cc: Mark Allen/Dallas/IBM@IBMUS > Date: 04/04/2014 09:49 AM > Subject: Re: Fw: romio questions > > ------------------------------------------------------------------------ > > > > > > On 04/04/2014 01:22 AM, Paul Coffman wrote: > > > ----- Forwarded by Paul Coffman/Rochester/IBM on 04/04/2014 01:22 AM > ----- > > > > From: Mark Allen/Dallas/IBM > > To: Paul Coffman/Rochester/IBM@IBMUS, > > Date: 04/04/2014 01:07 AM > > Subject: romio questions > > ------------------------------------------------------------------------ > > > > > > I have two questions/topics for you: > > > > First I wanted to ask do you happen to know if romio is thread safe? I > > see a fair number of critical-section begin/end macros and am guessing > > it is, but thought I'd ask anyway. > > the internal romio routines (things in the ADIO and ADIOI namespace) > rely on several bits of global state -- the flattened representation of > the file and memory datatype come to mind first, but there are probably > others. The critical section macros are at the MPI-IO interface to > romio and should provide a "big lock" to the rest. > > I haven't tried this though. I would be a little nervous about it > working without a few patches. > > > Second, I noticed romio uses an extension of generalized requests > > described here: > > > http://www.cs.illinois.edu/~wgropp/bib/papers/2007/latham_grequest-enhance-… > > > > but the code looks confused on whether the proposed wait_fn callback is > > waiting for a single request or all the requests. > > > > In romio's definitions of > > ADIOI_PVFS2_aio_wait_fn > > ADIOI_GEN_aio_wait_fn > > the wait_fn looks to me very much like it's waiting for all the reqs, vs > > the NTFS function that looks like it's waiting on some > > ADIOI_NTFS_aio_wait_fn > > the intent for the extended wait_fn was indeed to wait for all > outstanding generalized requests. Specifically, to call aio_suspend on > more than one operation at a time. > > I would not read too much into NTFS_aio_wait_fn. that work was done > quickly for the paper and then set aside. > > > And where these wait_fn callbacks get used, MPI_Waitsome uses > > MPIR_Grequest_progress_poke which conditionally calls wait_fn. It seems > > to me this would make MPI_Waitsome erroneously block until all its > > generalized requests finish (the code in MPIR_Grequest_progress_poke > > looks like it believes the wait_fn is supposed to just complete one > > request). > > It has been some years since I looked at the waitsome/extended-grequest > interaction, but it does sound like I could have implemented it better... > > > Is this an area you've worked with any or something we need to worry > > about? The reason I was looking at it was in order to pull in the new > > romio I figured I'd just add MPICH's concept of extended generalized > > requests into our MPI, so I wanted to make sure I understood how they > > were intended to work in mpich and I think it's pretty muddled there > > unless I'm reading it wrong. > > You would be the first developers besides me to look closely at our > extended generalized request proposal. It's no surprise to me it's > muddled, since it hasn't had a lot of attention over the years. > > at various points in the last 6 years MPICH developers have modified the > implementation of generalized requests to keep the common code paths > speedy. That might explain some murkiness or things like 'waitsome' > actually waiting for everything. > > ==rob > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0