January 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] single node mpiio
by Jaln 12 Jan '16

12 Jan '16

ah, Thanks Wei-keng, I'll try that. Best, Jialin On Tue, Jan 12, 2016 at 11:59 AM, Wei-keng Liao < wkliao(a)eecs.northwestern.edu> wrote: > Hi, Rob, > > I just want to clarify the locking behavior on Lustre (at least from my > understanding). The 3 independent I/O requests coming from the same compute > node will be considered by the Lustre as requests from the same client > (because I/O are system calls and all 3 processes run on the same OS). > Thus, no lock granting or revoking will occur within all processes in > the same node. > > The three independent requests appear to run one after another is because > (as you explained) there is only one link from that node to the Lustre > system. This is probably the system behavior. Jialin, you might want to > verify this by writing a simple MPI program using POSIX write calls. > > > Wei-keng > > On Jan 12, 2016, at 1:22 PM, Rob Latham wrote: > > > > > > > On 01/12/2016 11:36 AM, Jaln wrote: > >> Hi, > >> I'm running hdft with mpiio on a single compute node (32 cores), 10 > >> OSTs, the file system is lustre v2.5. > >> I submit the job with 3 processes, they are writing to a shared file, > >> which is about 3GBs, > >> and each process writes 1/3 of the file, for example, > >> The array is a 4D double array, 3*32*1024*128, then each process writes > >> 32*1024*128 to the file, which is contiguous. > >> > >> I observed some wired performance number, I tried both independent I/O > >> and collective IO. > >> In the case of independent I/O, each rank seems to block each other and > >> finish writing one after another. But in collective I/O, all three ranks > >> reports same I/O cost, I think this is because there is only one > >> aggregator. > >> My question is, in the case of independent I/O, are the I/Os blocking > >> when accessing the file? > >> If not blocking, can I expect linear speedup on a single node by > >> increasing number of processes? > > > > There is a lot going on under the hood in collective mode. Let's start > with independent mode. Here's my mental model of what's going on: > > > > - Your 3 process application issues 3 lustre writes. > > - One of those will win the race to the file system, acquire a > whole-file lock, and begin to write. > > - Now the request from the 2nd place process arrives. Lustre will > revoke the whole-file lock and issue a lock for the beginning of the > request to the end of the file. > > - When the last process arrives, Lustre will revoke locks yet again, > and issue a lock from the beginning of the request to the end of the file. > > > > So yep, there is indeed a lot of blocking going on. It's not a formal > queue, more of a side effect of waiting for the Lustre MDS to issue the > required locks. > > > > In collective I/O (which I'll assume is a relatively recent ROMIO > implementation), one of your three mpi processes will be the aggregator. > You seem to know about two-phase collective I/O so I will make the rest of > the explanation brief. Yes, that aggregator will receive data from the > other proceses and issue all I/O. The other processes will wait until the > aggregator finishes, which is why you see all processes reporting the same > run time for I/O. > > > > You are unlikely to get speedup from a single node for two main > reasons: There are the aforementioned lock contention issues. There is a > single link from that node to the 10 Lustre OSTs. > > > > In fact, due to the lock revocation activity I described above, you are > likely to see a massive performance crash when you involve a second node. > Then, as you add more nodes, you will finally light up enough network links > to see a speedup. > > > > ==rob > > > >> > >> Best, > >> Jialin > >> Lawrence Berkeley Lab > >> > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Genius only means hard-working all one's life _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] single node mpiio
by Jaln 12 Jan '16

12 Jan '16

Thanks Rob, What is the reason of the whole file lock? is it due the bug in lustre 2.5? which has been fixed in 2.6 and 2.7. Or is it just due to such I/O pattern? If without file-locking, is the "single link" to the 10 osts actually blocking the I/O from being true parallel I/O? Best, Jialin Lawrence Berkeley Lab On Tue, Jan 12, 2016 at 11:22 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 01/12/2016 11:36 AM, Jaln wrote: > >> Hi, >> I'm running hdft with mpiio on a single compute node (32 cores), 10 >> OSTs, the file system is lustre v2.5. >> I submit the job with 3 processes, they are writing to a shared file, >> which is about 3GBs, >> and each process writes 1/3 of the file, for example, >> The array is a 4D double array, 3*32*1024*128, then each process writes >> 32*1024*128 to the file, which is contiguous. >> >> I observed some wired performance number, I tried both independent I/O >> and collective IO. >> In the case of independent I/O, each rank seems to block each other and >> finish writing one after another. But in collective I/O, all three ranks >> reports same I/O cost, I think this is because there is only one >> aggregator. >> My question is, in the case of independent I/O, are the I/Os blocking >> when accessing the file? >> If not blocking, can I expect linear speedup on a single node by >> increasing number of processes? >> > > There is a lot going on under the hood in collective mode. Let's start > with independent mode. Here's my mental model of what's going on: > > - Your 3 process application issues 3 lustre writes. > - One of those will win the race to the file system, acquire a whole-file > lock, and begin to write. > - Now the request from the 2nd place process arrives. Lustre will revoke > the whole-file lock and issue a lock for the beginning of the request to > the end of the file. > - When the last process arrives, Lustre will revoke locks yet again, and > issue a lock from the beginning of the request to the end of the file. > > So yep, there is indeed a lot of blocking going on. It's not a formal > queue, more of a side effect of waiting for the Lustre MDS to issue the > required locks. > > In collective I/O (which I'll assume is a relatively recent ROMIO > implementation), one of your three mpi processes will be the aggregator. > You seem to know about two-phase collective I/O so I will make the rest of > the explanation brief. Yes, that aggregator will receive data from the > other proceses and issue all I/O. The other processes will wait until the > aggregator finishes, which is why you see all processes reporting the same > run time for I/O. > > You are unlikely to get speedup from a single node for two main reasons: > There are the aforementioned lock contention issues. There is a single > link from that node to the 10 Lustre OSTs. > > In fact, due to the lock revocation activity I described above, you are > likely to see a massive performance crash when you involve a second node. > Then, as you add more nodes, you will finally light up enough network links > to see a speedup. > > ==rob > > >> Best, >> Jialin >> Lawrence Berkeley Lab >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Genius only means hard-working all one's life _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] single node mpiio
by Wei-keng Liao 12 Jan '16

12 Jan '16

Hi, Rob, I just want to clarify the locking behavior on Lustre (at least from my understanding). The 3 independent I/O requests coming from the same compute node will be considered by the Lustre as requests from the same client (because I/O are system calls and all 3 processes run on the same OS). Thus, no lock granting or revoking will occur within all processes in the same node. The three independent requests appear to run one after another is because (as you explained) there is only one link from that node to the Lustre system. This is probably the system behavior. Jialin, you might want to verify this by writing a simple MPI program using POSIX write calls. Wei-keng On Jan 12, 2016, at 1:22 PM, Rob Latham wrote: > > > On 01/12/2016 11:36 AM, Jaln wrote: >> Hi, >> I'm running hdft with mpiio on a single compute node (32 cores), 10 >> OSTs, the file system is lustre v2.5. >> I submit the job with 3 processes, they are writing to a shared file, >> which is about 3GBs, >> and each process writes 1/3 of the file, for example, >> The array is a 4D double array, 3*32*1024*128, then each process writes >> 32*1024*128 to the file, which is contiguous. >> >> I observed some wired performance number, I tried both independent I/O >> and collective IO. >> In the case of independent I/O, each rank seems to block each other and >> finish writing one after another. But in collective I/O, all three ranks >> reports same I/O cost, I think this is because there is only one >> aggregator. >> My question is, in the case of independent I/O, are the I/Os blocking >> when accessing the file? >> If not blocking, can I expect linear speedup on a single node by >> increasing number of processes? > > There is a lot going on under the hood in collective mode. Let's start with independent mode. Here's my mental model of what's going on: > > - Your 3 process application issues 3 lustre writes. > - One of those will win the race to the file system, acquire a whole-file lock, and begin to write. > - Now the request from the 2nd place process arrives. Lustre will revoke the whole-file lock and issue a lock for the beginning of the request to the end of the file. > - When the last process arrives, Lustre will revoke locks yet again, and issue a lock from the beginning of the request to the end of the file. > > So yep, there is indeed a lot of blocking going on. It's not a formal queue, more of a side effect of waiting for the Lustre MDS to issue the required locks. > > In collective I/O (which I'll assume is a relatively recent ROMIO implementation), one of your three mpi processes will be the aggregator. You seem to know about two-phase collective I/O so I will make the rest of the explanation brief. Yes, that aggregator will receive data from the other proceses and issue all I/O. The other processes will wait until the aggregator finishes, which is why you see all processes reporting the same run time for I/O. > > You are unlikely to get speedup from a single node for two main reasons: There are the aforementioned lock contention issues. There is a single link from that node to the 10 Lustre OSTs. > > In fact, due to the lock revocation activity I described above, you are likely to see a massive performance crash when you involve a second node. Then, as you add more nodes, you will finally light up enough network links to see a speedup. > > ==rob > >> >> Best, >> Jialin >> Lawrence Berkeley Lab >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] single node mpiio
by Rob Latham 12 Jan '16

12 Jan '16

On 01/12/2016 11:36 AM, Jaln wrote: > Hi, > I'm running hdft with mpiio on a single compute node (32 cores), 10 > OSTs, the file system is lustre v2.5. > I submit the job with 3 processes, they are writing to a shared file, > which is about 3GBs, > and each process writes 1/3 of the file, for example, > The array is a 4D double array, 3*32*1024*128, then each process writes > 32*1024*128 to the file, which is contiguous. > > I observed some wired performance number, I tried both independent I/O > and collective IO. > In the case of independent I/O, each rank seems to block each other and > finish writing one after another. But in collective I/O, all three ranks > reports same I/O cost, I think this is because there is only one > aggregator. > My question is, in the case of independent I/O, are the I/Os blocking > when accessing the file? > If not blocking, can I expect linear speedup on a single node by > increasing number of processes? There is a lot going on under the hood in collective mode. Let's start with independent mode. Here's my mental model of what's going on: - Your 3 process application issues 3 lustre writes. - One of those will win the race to the file system, acquire a whole-file lock, and begin to write. - Now the request from the 2nd place process arrives. Lustre will revoke the whole-file lock and issue a lock for the beginning of the request to the end of the file. - When the last process arrives, Lustre will revoke locks yet again, and issue a lock from the beginning of the request to the end of the file. So yep, there is indeed a lot of blocking going on. It's not a formal queue, more of a side effect of waiting for the Lustre MDS to issue the required locks. In collective I/O (which I'll assume is a relatively recent ROMIO implementation), one of your three mpi processes will be the aggregator. You seem to know about two-phase collective I/O so I will make the rest of the explanation brief. Yes, that aggregator will receive data from the other proceses and issue all I/O. The other processes will wait until the aggregator finishes, which is why you see all processes reporting the same run time for I/O. You are unlikely to get speedup from a single node for two main reasons: There are the aforementioned lock contention issues. There is a single link from that node to the 10 Lustre OSTs. In fact, due to the lock revocation activity I described above, you are likely to see a massive performance crash when you involve a second node. Then, as you add more nodes, you will finally light up enough network links to see a speedup. ==rob > > Best, > Jialin > Lawrence Berkeley Lab > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI I/O and Windows
by Rob Latham 12 Jan '16

12 Jan '16

On 01/12/2016 11:17 AM, Eric Chamberland wrote: > Otherwise, we would have to implement all of > > MPI_Get_address(x,y) > MPI_File_write_at(a,b,c,d,e,f) > MPI_File_read_at_all(a,b,c,d,e,f) > MPI_File_write_all_begin(a,b,c,d) > MPI_File_write_all_end(a,b,c) > MPI_File_write(a,b,c,d,e) > MPI_File_read_all_begin(a,b,c,d) > MPI_File_read_all_end(a,b,c) > MPI_FILE_delete > MPI_Info_set > MPI_Info_free > MPI_Info_create yep, those all look like routines MS-MPI and Intel MPI would support. nothing unusual there. > to work in sequential, which is feasible too... > >> I don't know what the consistency semantics of SMB/CFS are: when are >> writes from one client visible to another? >> > > Can't tell... when I used windows for the last time I was so younger! :) as this list is for mpich, and mpich no longer has window support, I'll have to encourage you to contact MS-MPI or Intel MPI folks for follow up questions. ==rob > > Thanks for the answer! > > Eric > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

single node mpiio
by Jaln 12 Jan '16

12 Jan '16

Hi, I'm running hdft with mpiio on a single compute node (32 cores), 10 OSTs, the file system is lustre v2.5. I submit the job with 3 processes, they are writing to a shared file, which is about 3GBs, and each process writes 1/3 of the file, for example, The array is a 4D double array, 3*32*1024*128, then each process writes 32*1024*128 to the file, which is contiguous. I observed some wired performance number, I tried both independent I/O and collective IO. In the case of independent I/O, each rank seems to block each other and finish writing one after another. But in collective I/O, all three ranks reports same I/O cost, I think this is because there is only one aggregator. My question is, in the case of independent I/O, are the I/Os blocking when accessing the file? If not blocking, can I expect linear speedup on a single node by increasing number of processes? Best, Jialin Lawrence Berkeley Lab _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI I/O and Windows
by Eric Chamberland 12 Jan '16

12 Jan '16

Le 2016-01-12 10:46, Rob Latham a écrit : > > > On 01/11/2016 10:13 AM, Eric Chamberland wrote: >> Hi, >> >> is it possible to have MPI code that uses MPI I/O functions works under >> windows? >> >> Which version (package) of MPI can we try for that? > > definitely possible. MS-MPI implements the MPI-IO routines as does > Intel MPI. I don't have any experience with parallel I/O on Windows, > but I think you're going to find file system client support a bigger > challenge than MPI-IO support. > :) We are definitely binded to MPI I/O functions to read/write even in sequential mode... We are then happy to look for a zero lines of code solution to support windows for our software... Otherwise, we would have to implement all of MPI_Get_address(x,y) MPI_File_write_at(a,b,c,d,e,f) MPI_File_read_at_all(a,b,c,d,e,f) MPI_File_write_all_begin(a,b,c,d) MPI_File_write_all_end(a,b,c) MPI_File_write(a,b,c,d,e) MPI_File_read_all_begin(a,b,c,d) MPI_File_read_all_end(a,b,c) MPI_FILE_delete MPI_Info_set MPI_Info_free MPI_Info_create to work in sequential, which is feasible too... > I don't know what the consistency semantics of SMB/CFS are: when are > writes from one client visible to another? > Can't tell... when I used windows for the last time I was so younger! :) Thanks for the answer! Eric _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI I/O and Windows
by Rob Latham 12 Jan '16

12 Jan '16

On 01/11/2016 10:13 AM, Eric Chamberland wrote: > Hi, > > is it possible to have MPI code that uses MPI I/O functions works under > windows? > > Which version (package) of MPI can we try for that? definitely possible. MS-MPI implements the MPI-IO routines as does Intel MPI. I don't have any experience with parallel I/O on Windows, but I think you're going to find file system client support a bigger challenge than MPI-IO support. I don't know what the consistency semantics of SMB/CFS are: when are writes from one client visible to another? ==rob _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI I/O and Windows
by Eric Chamberland 11 Jan '16

11 Jan '16

Hi, is it possible to have MPI code that uses MPI I/O functions works under windows? Which version (package) of MPI can we try for that? Thanks, Eric _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] valgrind warning in MPIR_Process_status
by VandeVondele Joost 11 Jan '16

11 Jan '16

I can confirm that Daniel's patch also fixes the issue I was seeing. Thanks for the quick resolution. ________________________________ ________________________________ From: Jeff Hammond [jeff.science(a)gmail.com] Sent: Monday, January 11, 2016 6:51 AM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] valgrind warning in MPIR_Process_status To be explicit, the other thread is http://lists.mpich.org/pipermail/devel/2016-January/000677.html and it is indeed the same issue. I pushed dibanez-helper_fns-valgrind to mpich-review just now. https://trac.mpich.org/projects/mpich/ticket/2328 was created for tracking purposes. Jeff On Sun, Jan 10, 2016 at 5:04 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: I think this is the same issue reported on the devel list yesterday. If it is, Daniel Ibanez provided a patch that I've merged into my (completely unofficial fork on github) and will push to the official MPICH review repo tonight. If you want to try and/or verify Daniel's patch, see github/jeffhammond/mpich. The branch you want should be obvious from the name (I can't remember what it is off hand). Best, Jeff On Sunday, January 10, 2016, VandeVondele Joost <joost.vandevondele(a)mat.ethz.ch<mailto:[email protected]>> wrote: Hi, we upgraded from mpich 3.1.2 to 3.2 and our nightly builds under valgrind show a new valgrind warning, which I suppose originates from the mpi library, rather than our code (given the stack trace below and the fact that they are absent with 3.1): There are two slightly distinct traces (alltoall and alltoallv): ==30165== Conditional jump or move depends on uninitialised value(s) ==30165== at 0x6E00056: MPIR_Process_status (mpiimpl.h:4394) ==30165== by 0x6E00056: MPIC_Waitall (helper_fns.c:774) ==30165== by 0x6D5A853: MPIR_Alltoall_intra (alltoall.c:369) ==30165== by 0x6D5B63C: MPIR_Alltoall (alltoall.c:564) ==30165== by 0x6D5B6EF: MPIR_Alltoall_impl (alltoall.c:599) ==30165== by 0x6D5BF07: PMPI_Alltoall (alltoall.c:722) ==30165== by 0x6AFF0DC: PMPI_ALLTOALL (alltoallf.c:276) ==30165== by 0x1D04699: __message_passing_MOD_mp_alltoall_i22 ==30201== Conditional jump or move depends on uninitialised value(s) ==30201== at 0x6E00056: MPIR_Process_status (mpiimpl.h:4394) ==30201== by 0x6E00056: MPIC_Waitall (helper_fns.c:774) ==30201== by 0x6D5C4EA: MPIR_Alltoallv_intra (alltoallv.c:194) ==30201== by 0x6D5CEC9: MPIR_Alltoallv (alltoallv.c:339) ==30201== by 0x6D5CFAA: MPIR_Alltoallv_impl (alltoallv.c:376) ==30201== by 0x6D5D8CA: PMPI_Alltoallv (alltoallv.c:527) ==30201== by 0x6AFF13E: PMPI_ALLTOALLV (alltoallvf.c:276) ==30201== by 0x1CCE180: __message_passing_MOD_mp_alltoall_z22v (message_passing_z.f90:223) Is this issue already known/fixed ? I saw there was a somewhat related change (which is part of 3.2): https://trac.mpich.org/projects/mpich/changeset/be7c17b3177ccb0ede49761b44a… but maybe that change was not fully complete ? Best regards, Joost VandeVondele ________________________________ Prof. Dr. Joost VandeVondele Nanoscale Simulations ETH Zurich HIT G 41.4 Wolfgang-Pauli-Strasse 27 8093 Zurich, Switzerland Joost.VandeVondele(a)mat.ethz.ch http://www.nanosim.mat.ethz.ch/ Tel.: +41 (0)44 6338451 ________________________________ -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/ -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0