Every time the MPICH stack is “stuck” in the progress engine (i.e., nothing happened for some time), it’ll release the lock so other threads can grab it. So unless File_write_all is blocking on a file-system operation, other threads should be able to make progress just fine. — Pavan On Apr 4, 2014, at 4:34 PM, Rob Latham <[email protected]> wrote:
This is getting out of my depth. MPICH guys: Mark Allen is updating Platform's MPI and has been asking me some questions about multi-threaded ROMIO.
The independent ROMIO routines might make some local MPI calls to process the data type, but otherwise, yes, the only blocking call would be the file system operation.
I haven't thought about what would happen if MPI_File_write_all was called from two threads.
MPICH guys: there's no way the CS_ENTER/EXIT macros can be that clever, right?
==rob
On 04/04/2014 04:28 PM, Mark Allen wrote:
Thanks. I looked a bit at the CS_ENTER/EXIT code. Am I right that all the non-collective MPIO calls like MPI_File_read etc are only blocking in the sense of waiting on some local operation to complete? If that's the case it would be okay for them to hold a lock from beginning to end of the call like that. But for the collective MPIO operations I don't think you could just hold the lock for the entire function call since you might end up waiting for remote peers to arrive, and if two collectives were taking place at the same time they could mismatch and deadlock. But I'm not sure if that's what the code is doing or not. Is there a release of the lock hidden somewhere when collectives block? If inside the MPI_Allgather for example MPI performed a release and regain of the MPIO lock is the blocking MPIO collective holding any shared resource that would get corrupted by letting other threads in at that point?
Fwiw I also noticed mpi-io/ioreq_f2c.c looks to be a case where an early return runs the risk of a mismatched CS_ENTER w/o a corresponding CS_EXIT. It looks like all the other files used goto fn_exit to ensure a match.
Mark
Inactive hide details for Rob Latham ---04/04/2014 09:49:00 AM---On 04/04/2014 01:22 AM, Paul Coffman wrote: > ----- Forwarded Rob Latham ---04/04/2014 09:49:00 AM---On 04/04/2014 01:22 AM, Paul Coffman wrote:
----- Forwarded by Paul Coffman/Rochester/IBM on 04/04
From: Rob Latham <[email protected]> To: Paul Coffman/Rochester/IBM@IBMUS Cc: Mark Allen/Dallas/IBM@IBMUS Date: 04/04/2014 09:49 AM Subject: Re: Fw: romio questions
------------------------------------------------------------------------
On 04/04/2014 01:22 AM, Paul Coffman wrote:
----- Forwarded by Paul Coffman/Rochester/IBM on 04/04/2014 01:22 AM
From: Mark Allen/Dallas/IBM To: Paul Coffman/Rochester/IBM@IBMUS, Date: 04/04/2014 01:07 AM Subject: romio questions ------------------------------------------------------------------------
I have two questions/topics for you:
First I wanted to ask do you happen to know if romio is thread safe? I see a fair number of critical-section begin/end macros and am guessing it is, but thought I'd ask anyway.
the internal romio routines (things in the ADIO and ADIOI namespace) rely on several bits of global state -- the flattened representation of the file and memory datatype come to mind first, but there are probably others. The critical section macros are at the MPI-IO interface to romio and should provide a "big lock" to the rest.
I haven't tried this though. I would be a little nervous about it working without a few patches.
Second, I noticed romio uses an extension of generalized requests described here:
http://www.cs.illinois.edu/~wgropp/bib/papers/2007/latham_grequest-enhance-4...
but the code looks confused on whether the proposed wait_fn callback is waiting for a single request or all the requests.
In romio's definitions of ADIOI_PVFS2_aio_wait_fn ADIOI_GEN_aio_wait_fn the wait_fn looks to me very much like it's waiting for all the reqs, vs the NTFS function that looks like it's waiting on some ADIOI_NTFS_aio_wait_fn
the intent for the extended wait_fn was indeed to wait for all outstanding generalized requests. Specifically, to call aio_suspend on more than one operation at a time.
I would not read too much into NTFS_aio_wait_fn. that work was done quickly for the paper and then set aside.
And where these wait_fn callbacks get used, MPI_Waitsome uses MPIR_Grequest_progress_poke which conditionally calls wait_fn. It seems to me this would make MPI_Waitsome erroneously block until all its generalized requests finish (the code in MPIR_Grequest_progress_poke looks like it believes the wait_fn is supposed to just complete one request).
It has been some years since I looked at the waitsome/extended-grequest interaction, but it does sound like I could have implemented it better...
Is this an area you've worked with any or something we need to worry about? The reason I was looking at it was in order to pull in the new romio I figured I'd just add MPICH's concept of extended generalized requests into our MPI, so I wanted to make sure I understood how they were intended to work in mpich and I think it's pretty muddled there unless I'm reading it wrong.
You would be the first developers besides me to look closely at our extended generalized request proposal. It's no surprise to me it's muddled, since it hasn't had a lot of attention over the years.
at various points in the last 6 years MPICH developers have modified the implementation of generalized requests to keep the common code paths speedy. That might explain some murkiness or things like 'waitsome' actually waiting for everything.
==rob
-- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA
-- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA