Thanks Rob, What is the reason of the whole file lock? is it due the bug in lustre 2.5? which has been fixed in 2.6 and 2.7. Or is it just due to such I/O pattern? If without file-locking, is the "single link" to the 10 osts actually blocking the I/O from being true parallel I/O? Best, Jialin Lawrence Berkeley Lab On Tue, Jan 12, 2016 at 11:22 AM, Rob Latham <[email protected]> wrote:
On 01/12/2016 11:36 AM, Jaln wrote:
Hi, I'm running hdft with mpiio on a single compute node (32 cores), 10 OSTs, the file system is lustre v2.5. I submit the job with 3 processes, they are writing to a shared file, which is about 3GBs, and each process writes 1/3 of the file, for example, The array is a 4D double array, 3*32*1024*128, then each process writes 32*1024*128 to the file, which is contiguous.
I observed some wired performance number, I tried both independent I/O and collective IO. In the case of independent I/O, each rank seems to block each other and finish writing one after another. But in collective I/O, all three ranks reports same I/O cost, I think this is because there is only one aggregator. My question is, in the case of independent I/O, are the I/Os blocking when accessing the file? If not blocking, can I expect linear speedup on a single node by increasing number of processes?
There is a lot going on under the hood in collective mode. Let's start with independent mode. Here's my mental model of what's going on:
- Your 3 process application issues 3 lustre writes. - One of those will win the race to the file system, acquire a whole-file lock, and begin to write. - Now the request from the 2nd place process arrives. Lustre will revoke the whole-file lock and issue a lock for the beginning of the request to the end of the file. - When the last process arrives, Lustre will revoke locks yet again, and issue a lock from the beginning of the request to the end of the file.
So yep, there is indeed a lot of blocking going on. It's not a formal queue, more of a side effect of waiting for the Lustre MDS to issue the required locks.
In collective I/O (which I'll assume is a relatively recent ROMIO implementation), one of your three mpi processes will be the aggregator. You seem to know about two-phase collective I/O so I will make the rest of the explanation brief. Yes, that aggregator will receive data from the other proceses and issue all I/O. The other processes will wait until the aggregator finishes, which is why you see all processes reporting the same run time for I/O.
You are unlikely to get speedup from a single node for two main reasons: There are the aforementioned lock contention issues. There is a single link from that node to the 10 Lustre OSTs.
In fact, due to the lock revocation activity I described above, you are likely to see a massive performance crash when you involve a second node. Then, as you add more nodes, you will finally light up enough network links to see a speedup.
==rob
Best, Jialin Lawrence Berkeley Lab
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
-- Genius only means hard-working all one's life _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss