RE: alignment hints for calculating the file domain.
Hi, All, Wangdi has observed a lot of lock contentions when the file domain partitions in collective IO are not aligned on Lustre stripe boundary (say 1MB). As a remedy of that, we are proposing a minor change on the implementation of ADIOI_GEN_Calc_file_domains() for it to respect an advisory alignment. The attached patch illustrates what we are proposing. We hope this can get incorporated upstream in future release of romio. It would save the need of a modified ADIOI_GEN_Calc_file_domains() in the ADIO driver for file system like Lustre, and allow us to feed ADIOI_GEN_Calc_file_domains() with a preferred alignment. Please consider. Thanks, Weikuan P.S.: We will use [email protected] for further discussions if mpich2-dev is not for this purpose.
-----Original Message----- From: wangdi [mailto:[email protected]] Sent: Tuesday, July 10, 2007 12:33 AM To: [email protected]; Yu, Weikuan Subject: alignment hints for calculating the file domain.
Hello,
We are trying to improve lustre adio driver currently for mpich2. We found that all the file system use ADIOI_GEN_WriteStridedColl as their collective write API in MPICH2. In this API, it will calculate the file domains and distribute the I/O segment evenly around all the clients according to the file domain size. And, when calculating the file domains, it just divide the whole i/o area(min, max) by the count of the clients, but did not take into account the specifics of some file systems, which might be sensitive to this file_domain_size. For example, some file system might need special alignment for better performance instead of this evenly divided domain_size.
So could you export a new API (ADIOI_Calc_file_domains) here, then those file systems, which are not sensitive to the alignment, just call current calc_file_domains API(might be renamed as ADIO_GEN_Calc_file_domains), and for those file systems, which need the special alignment could get a alignment hint before real calculation. Maybe the calc file domain API could be changed in this way?
ADIOI_XXX_Calc_file_domains() { int alignment = get_xxx_alignment(); *fd_size_ptr = alignment;
ADIOI_GEN_Calc_file_domains(......, fd_size_ptr);
}
ADIOI_GEN_Calc_file_domains(......, *fd_size) he { int alignment = *fd_size_ptr;
........
fd_size = ((max_end_offset - min_st_offset + 1) + nprocs_for_coll - 1)/nprocs_for_coll;
if (alignment != 0) { fd_size = ((fd_size + alignmentan - 1) / alignment) * alignment; }
........
*fd_size_ptr = fd_size;
}
Thanks
-- Regards, Tom Wangdi -- Cluster File Systems, Inc Software Engineer http://www.clusterfs.com
On Tue, Jul 10, 2007 at 08:55:18AM -0400, Yu, Weikuan wrote:
Wangdi has observed a lot of lock contentions when the file domain partitions in collective IO are not aligned on Lustre stripe boundary (say 1MB). As a remedy of that, we are proposing a minor change on the implementation of ADIOI_GEN_Calc_file_domains() for it to respect an advisory alignment. The attached patch illustrates what we are proposing.
Ok, so you turn fd_size from an output parameter to an in/out parameter. I don't think that's introducing noticeably more complexity to the interface. Before you call calc_file_domains, how will you set fd_size? Through a hint the user can override?
P.S.: We will use [email protected] for further discussions if mpich2-dev is not for this purpose.
I think this list will be fine. ==rob -- Rob Latham Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF Argonne National Lab, IL USA B29D F333 664A 4280 315B
-----Original Message----- From: Robert Latham [mailto:[email protected]]
On Tue, Jul 10, 2007 at 08:55:18AM -0400, Yu, Weikuan wrote:
Wangdi has observed a lot of lock contentions when the file domain partitions in collective IO are not aligned on Lustre stripe boundary (say 1MB). As a remedy of that, we are proposing a minor change on the implementation of ADIOI_GEN_Calc_file_domains() for it to respect an advisory alignment. The attached patch illustrates what we are proposing.
Ok, so you turn fd_size from an output parameter to an in/out parameter. I don't think that's introducing noticeably more complexity to the interface.
Right, that's the intention. We want to avoid a new function which adds very little to the existing one.
Before you call calc_file_domains, how will you set fd_size? Through a hint the user can override?
We can do that with a separate ADIO driver. For Lustre, we can find out the stripe_size from the file attributes. Users can also suggest an hint on stripe_size of new files. --Weikuan
P.S.: We will use [email protected] for further discussions if mpich2-dev is not for this purpose.
I think this list will be fine.
==rob
-- Rob Latham Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF Argonne National Lab, IL USA B29D F333 664A 4280 315B
Further update on this, attached is the working code I have in the Lustre ADIO driver. The condition checking on "fd->file_system == ADIO_LUSTRE" can be removed for generic usage, assuming fs-specific ADIO saves a file's striping parameters in fd->info. -- Weikuan Yu <+> 1-865-574-7990 http://ft.ornl.gov/~wyu/ -----Original Message----- From: [email protected] on behalf of Yu, Weikuan Sent: Tue 7/10/2007 11:00 AM To: Robert Latham Cc: wangdi; [email protected] Subject: RE: [MPICH2-dev] RE: alignment hints for calculating the file domain.
-----Original Message----- From: Robert Latham [mailto:[email protected]]
On Tue, Jul 10, 2007 at 08:55:18AM -0400, Yu, Weikuan wrote:
Wangdi has observed a lot of lock contentions when the file domain partitions in collective IO are not aligned on Lustre stripe boundary (say 1MB). As a remedy of that, we are proposing a minor change on the implementation of ADIOI_GEN_Calc_file_domains() for it to respect an advisory alignment. The attached patch illustrates what we are proposing.
Ok, so you turn fd_size from an output parameter to an in/out parameter. I don't think that's introducing noticeably more complexity to the interface.
Right, that's the intention. We want to avoid a new function which adds very little to the existing one.
Before you call calc_file_domains, how will you set fd_size? Through a hint the user can override?
We can do that with a separate ADIO driver. For Lustre, we can find out the stripe_size from the file attributes. Users can also suggest an hint on stripe_size of new files. --Weikuan
P.S.: We will use [email protected] for further discussions if mpich2-dev is not for this purpose.
I think this list will be fine.
==rob
-- Rob Latham Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF Argonne National Lab, IL USA B29D F333 664A 4280 315B
Before you call calc_file_domains, how will you set fd_size? Through a hint the user can override?
We can do that with a separate ADIO driver. For Lustre, we can find out the stripe_size from the file attributes. Users can also suggest an hint on stripe_size of new files.
To add a little more, this can end up a need of a Lustre-specific collective IO calls, which I think is the way we are considering. That said, the other suggestion from Wangdi would allow the reuse of current collective IO calls, though it would involve a little more changes. That direction would be good too if that's something you folks prefer. --Weikuan
participants (2)
-
robl＠mcs.anl.gov -
Yu, Weikuan