On 01/12/2016 11:36 AM, Jaln wrote:

Hi,
I'm running hdft with mpiio on a single compute node (32 cores), 10
OSTs, the file system is lustre v2.5.
I submit the job with 3 processes, they are writing to a shared file,
which is about 3GBs,
and each process writes 1/3 of the file, for example,
The array is a 4D double array, 3*32*1024*128, then each process writes
32*1024*128 to the file, which is contiguous.

I observed some wired performance number, I tried both independent I/O
and collective IO.
In the case of independent I/O, each rank seems to block each other and
finish writing one after another. But in collective I/O, all three ranks
reports same I/O cost, I think this is because there is only one
aggregator.
My question is, in the case of independent I/O, are the I/Os blocking
when accessing the file?
If not blocking, can I expect linear speedup on a single node by
increasing number of processes?

There is a lot going on under the hood in collective mode.  Let's start with independent mode.  Here's my mental model of what's going on:

- Your 3 process application issues 3 lustre writes.
- One of those will win the race to the file system, acquire a whole-file lock, and begin to write.
- Now the request from the 2nd place process arrives.  Lustre will revoke the whole-file lock and issue a lock for the beginning of the request to the end of the file.
- When the last process arrives, Lustre will revoke locks  yet again, and issue a lock from the beginning of the request to the end of the file.

So yep, there is indeed a lot of blocking going on.  It's not a formal queue, more of a side effect of waiting for the Lustre MDS to issue the required locks.

In collective I/O (which I'll assume is a relatively recent ROMIO implementation), one of your three mpi processes will be the aggregator.  You seem to know about two-phase collective I/O so I will make the rest of the explanation brief.  Yes, that aggregator will receive data from the other proceses and issue all I/O.   The other processes will wait until the aggregator finishes, which is why you see all processes reporting the same run time for I/O.

You are unlikely to get speedup from a single node for two main reasons:  There are the aforementioned lock contention issues.  There is a single link from that node to the 10 Lustre OSTs.

In fact, due to the lock revocation activity I described above, you are likely to see a massive performance crash when you involve a second node.  Then, as you add more nodes, you will finally light up enough network links to see a speedup.

==rob

Best,
Jialin
Lawrence Berkeley Lab


_______________________________________________
discuss mailing list     discuss@mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________
discuss mailing list     discuss@mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss