Hi, I'd like to reserve the usual time on Mon 3/16 to discuss our shared client API. We have about one quarter left to adopt a shared client API, and while we have a good start (the one we designed a long time ago) we all found some things that are not as nice or efficient as they could be. Some things just need to be clarified. Some examples: - the probe+read race condition/inefficiency. - can update id/version decrease? - is it valid to write a 0-version (or can only reset do that)? ... So, I'd like to dedicate 3/16, 3-4PM CST (usual concall number) to review the API. This will mainly be relevant for the Sirocco/Triton teams (as well as TAMU, who can share their experiences implementing the posix client on top of this API). For preparation, I would ask that everybody tries to prepare a list of issues we can go over in the call. Please confirm if you can make the call (which should be no problem, since if I recall correctly, we were going to keep this time open every week even if the call was only once a month). Thanks, Dries
Sorry, Mon 3/17. Dries * Dries Kimpe <[email protected]> [2014-03-05 18:34:07]:
Hi,
I'd like to reserve the usual time on Mon 3/16 to discuss our shared client API. We have about one quarter left to adopt a shared client API, and while we have a good start (the one we designed a long time ago) we all found some things that are not as nice or efficient as they could be. Some things just need to be clarified.
Some examples: - the probe+read race condition/inefficiency. - can update id/version decrease? - is it valid to write a 0-version (or can only reset do that)? ...
So, I'd like to dedicate 3/16, 3-4PM CST (usual concall number) to review the API.
This will mainly be relevant for the Sirocco/Triton teams (as well as TAMU, who can share their experiences implementing the posix client on top of this API).
For preparation, I would ask that everybody tries to prepare a list of issues we can go over in the call.
Please confirm if you can make the call (which should be no problem, since if I recall correctly, we were going to keep this time open every week even if the call was only once a month).
Thanks, Dries
_______________________________________________ ASG mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/asg
Reminder of our call to discuss shared client API today. Conference Access Number: 760-984-1000 Conference Passcode: 692481 # Dries * Dries Kimpe <[email protected]> [2014-03-06 14:03:41]:
Sorry, Mon 3/17.
Dries
* Dries Kimpe <[email protected]> [2014-03-05 18:34:07]:
Hi,
I'd like to reserve the usual time on Mon 3/16 to discuss our shared client API. We have about one quarter left to adopt a shared client API, and while we have a good start (the one we designed a long time ago) we all found some things that are not as nice or efficient as they could be. Some things just need to be clarified.
Some examples: - the probe+read race condition/inefficiency. - can update id/version decrease? - is it valid to write a 0-version (or can only reset do that)? ...
So, I'd like to dedicate 3/16, 3-4PM CST (usual concall number) to review the API.
This will mainly be relevant for the Sirocco/Triton teams (as well as TAMU, who can share their experiences implementing the posix client on top of this API).
For preparation, I would ask that everybody tries to prepare a list of issues we can go over in the call.
Please confirm if you can make the call (which should be no problem, since if I recall correctly, we were going to keep this time open every week even if the call was only once a month).
Thanks, Dries
_______________________________________________ ASG mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/asg
Notes from the Meeting: https://wiki.mcs.anl.gov/asg-private/index.php/ASG_API == Overview == Meeting on 2013-03-17 to discuss any current issues with the ASG API as proposed. The following topics were discussed. == Action Items == * Kevin - type up and send out notes * Phil - update asg.h to reflect API changes == Read + Probe == ==== Issue ==== * race of contents changing between probe and read ==== Discussion ==== * versions may change between probe and read * solve with batch that enforces isolation ** (Phil) ASG was system API but batch is only for per-server which alters API usage * one solution with no isolation is probe+read in a single command * local transaction for a single server might be useful for this * Can this be used to make a consistent duplicate? (cloning) ==== Decision ==== * Update asg_read call to add ‘probe for record’ parameters (may specify NULL if you don’t want the probe data) * No change needed to support multiple probes, just always start at n+1 from previous returned call ==== Issue ==== * how to handle need to make multiple probe calls, if you can't return all results in one call. ==== Decision ==== * no change needed ** always probe (n+1) ==== Decision ==== * Switch to “update_id” instead of version_id in ASG documentation == Multiple Round-Trips for Probes and Reads == ==== Issue ==== * a large discontiguous file could result in requiring many probe+reads ==== Discussion ==== * (Matt) use probe+read, don’t ask for probe results and zero-fill data (assume 1 byte records with zero value) * (Dries) have a method to request default record behavior (so for zero-fill, ask for default of 1-byte zero records) (requires storing this metadata with object/fork/etc) * (Dries) possible uses for security implementation ==== Decision ==== * none == Splitting File across multiple objects on different Servers == ==== Issue ==== * We likely need a way to split a large file across multiple objects on multiple servers. How do we handle updates safely across object boundaries? ==== Discussion ==== * do we need to support this within the storage system or can we force the client to handle it * does this really work in current file systems or do users really expect this to work? * Reddy will think more on this topic ==== Decision ==== * none == Can update_id decrease (or is it allowed to)? == ==== Issue ==== * Should we allow a decreasing update_id? ==== Discussion ==== * no one had a good reason to support this outside of reset ==== Decision ==== * do not allow reducing update_id, other than via reset and reset always goes to update_id = zero * May use same update_id on write * Zero is not a valid update_id to write == Security Capabilities == ==== Issue ==== * how to integrate security into ASG API ==== Discussion ==== * (Dries) separate APIs, just add single capability argument to all other APIs * tabled discussion for now since it is not needed at this time == Nesting of IDs in Sirocco == ==== Issue ==== * Sirocco allows generic nesting of ID spaces in the local store API, should this be extended to ASG? ==== Discussion ==== * any level of ID may store data * new API probe_child (which gives details about lower levels) * new API probe_data (which only gives data) ==== Decision ==== * don’t propose to use this in ASG API at this time * this could be part of local store API
Thanks. Notes inline. Apologies if I am asking naive questions — been a while since I last looked at the API I will admit. — Rob On Mar 17, 2014, at 5:10 PM, Harms, Kevin N. <[email protected]> wrote:
Notes from the Meeting:
https://wiki.mcs.anl.gov/asg-private/index.php/ASG_API
== Overview == Meeting on 2013-03-17 to discuss any current issues with the ASG API as proposed. The following topics were discussed.
== Action Items == * Kevin - type up and send out notes * Phil - update asg.h to reflect API changes
== Read + Probe == ==== Issue ==== * race of contents changing between probe and read ==== Discussion ==== * versions may change between probe and read * solve with batch that enforces isolation ** (Phil) ASG was system API but batch is only for per-server which alters API usage * one solution with no isolation is probe+read in a single command * local transaction for a single server might be useful for this * Can this be used to make a consistent duplicate? (cloning) ==== Decision ==== * Update asg_read call to add ‘probe for record’ parameters (may specify NULL if you don’t want the probe data) * No change needed to support multiple probes, just always start at n+1 from previous returned call ==== Issue ==== * how to handle need to make multiple probe calls, if you can't return all results in one call. ==== Decision ==== * no change needed ** always probe (n+1)
Is this describing a mechanism for iterating through results of a probe call? If so, is some sort of cookie or something passed around with the call that enables the state to be serialized and not require tracking of clients? Assumption is that since the probe is relative to some update_id, results don’t change (which would otherwise make things v. hard)?
==== Decision ==== * Switch to “update_id” instead of version_id in ASG documentation
== Multiple Round-Trips for Probes and Reads == ==== Issue ==== * a large discontiguous file could result in requiring many probe+reads ==== Discussion ==== * (Matt) use probe+read, don’t ask for probe results and zero-fill data (assume 1 byte records with zero value) * (Dries) have a method to request default record behavior (so for zero-fill, ask for default of 1-byte zero records) (requires storing this metadata with object/fork/etc) * (Dries) possible uses for security implementation ==== Decision ==== * none
Are we really proposing a fill capability on the object server side (and thus some semantic about filling records in order), or is this more about an ability to have the server create “virtual” records that enable us to make more efficient requests? Are we really happy with this probe+read approach? Seems like we’re driving down a path to multiple operations when we only want one. What am I missing?
== Splitting File across multiple objects on different Servers == ==== Issue ==== * We likely need a way to split a large file across multiple objects on multiple servers. How do we handle updates safely across object boundaries? ==== Discussion ==== * do we need to support this within the storage system or can we force the client to handle it * does this really work in current file systems or do users really expect this to work? * Reddy will think more on this topic ==== Decision ==== * none
It works fine on some current file systems. Not all file systems over time have obtained all locks for v. large writes, but I do not know current status for Lustre, etc. GPFS probably does behave correctly.
== Can update_id decrease (or is it allowed to)? == ==== Issue ==== * Should we allow a decreasing update_id? ==== Discussion ==== * no one had a good reason to support this outside of reset ==== Decision ==== * do not allow reducing update_id, other than via reset and reset always goes to update_id = zero * May use same update_id on write * Zero is not a valid update_id to write
So clients have to coordinate if writing to multiple update_ids concurrently, or they will get an error? Surely I am misunderstanding. What does it mean to “decrease” an update_id?
== Security Capabilities == ==== Issue ==== * how to integrate security into ASG API ==== Discussion ==== * (Dries) separate APIs, just add single capability argument to all other APIs * tabled discussion for now since it is not needed at this time
== Nesting of IDs in Sirocco == ==== Issue ==== * Sirocco allows generic nesting of ID spaces in the local store API, should this be extended to ASG? ==== Discussion ==== * any level of ID may store data * new API probe_child (which gives details about lower levels) * new API probe_data (which only gives data) ==== Decision ==== * don’t propose to use this in ASG API at this time * this could be part of local store API
_______________________________________________ ASG mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/asg
Hi Rob, Replies inline. On 3/18/14, 11:17 AM, "Rob Ross" <[email protected]> wrote:
Thanks. Notes inline. Apologies if I am asking naive questions ‹ been a while since I last looked at the API I will admit. ‹ Rob
On Mar 17, 2014, at 5:10 PM, Harms, Kevin N. <[email protected]> wrote:
Notes from the Meeting:
https://wiki.mcs.anl.gov/asg-private/index.php/ASG_API
== Overview == Meeting on 2013-03-17 to discuss any current issues with the ASG API as proposed. The following topics were discussed.
== Action Items == * Kevin - type up and send out notes * Phil - update asg.h to reflect API changes
== Read + Probe == ==== Issue ==== * race of contents changing between probe and read ==== Discussion ==== * versions may change between probe and read * solve with batch that enforces isolation ** (Phil) ASG was system API but batch is only for per-server which alters API usage * one solution with no isolation is probe+read in a single command * local transaction for a single server might be useful for this * Can this be used to make a consistent duplicate? (cloning) ==== Decision ==== * Update asg_read call to add Œprobe for record¹ parameters (may specify NULL if you don¹t want the probe data) * No change needed to support multiple probes, just always start at n+1 from previous returned call ==== Issue ==== * how to handle need to make multiple probe calls, if you can't return all results in one call. ==== Decision ==== * no change needed ** always probe (n+1)
Is this describing a mechanism for iterating through results of a probe call? If so, is some sort of cookie or something passed around with the call that enables the state to be serialized and not require tracking of clients? Assumption is that since the probe is relative to some update_id, results don¹t change (which would otherwise make things v. hard)?
This is describing how to iterate a probe when there are other servers inserting entries into the address space. The probe call does return the offset of the next record, but if another server were to insert something in the address space between the end of the last probe and the start of the next one, the server doing the probe would completely skip that insertion. It was decided that, if the server really cared about that use case, it could ignore the ³next offset² return value from the probe, and just probe from just beyond the end of the last interval it got. Can you clarify what you mean by ³probe is relative to some update_id²?
==== Decision ==== * Switch to ³update_id² instead of version_id in ASG documentation
== Multiple Round-Trips for Probes and Reads == ==== Issue ==== * a large discontiguous file could result in requiring many probe+reads ==== Discussion ==== * (Matt) use probe+read, don¹t ask for probe results and zero-fill data (assume 1 byte records with zero value) * (Dries) have a method to request default record behavior (so for zero-fill, ask for default of 1-byte zero records) (requires storing this metadata with object/fork/etc) * (Dries) possible uses for security implementation ==== Decision ==== * none
Are we really proposing a fill capability on the object server side (and thus some semantic about filling records in order), or is this more about an ability to have the server create ³virtual² records that enable us to make more efficient requests?
This one deserves a little motivation. I was concerned about doing file I/O in cases where objects have random writes and/or lots of holes. Just by virtue of incrementing update_ids, you get into a situation where the probe data may very well be larger than the data you¹re reading. There were two solutions that were briefly discussed, and one that wasn¹t: 1. (Admittedly the weakest) Use zero fill on the server side to obviate the need for an accompanying probe. Dries suggested the alternative of allowing the client to set the default record. A flag would be used to dictate whether fill could be used, and the server could send whichever representation would be smaller. (see item 3 below) 2. Allow probes to accept a flag that said that update_ids aren¹t important, so a single record could be used for dense data with various record IDs and homogeneous record size. Better, but does not cover the case where there are lots of holes. 3. (The one I didn¹t bring up yet, because it¹s weird.) Allow a single buffer space to contain probe and read contents. Having separate buffers can cause the probe buffer to fill well before the read, giving us a lot of data that doesn¹t have a proper map for detecting holes or differently sized records, necessitating a reread. This ties into probe+read, which I¹ll talk about next.
Are we really happy with this probe+read approach? Seems like we¹re driving down a path to multiple operations when we only want one. What am I missing?
Dries (and Lee, privately, about a month ago) proposed a combined probe+read operation that does both with one call. Dries¹s idea of the signature is similar to: int asg_probe_read ( asg_instance_t instance, asg_location_t location, asg_container_id_t container, asg_object_id_t object, asg_fork_id_t fork, asg_record_id_t start_record, asg_size_t recordcount, asg_flags_t flags, asg_version_t version_condition, void * readbuf, size_t readbufsize, asg_record_info_t * probebuf, size_t probebufsize, asg_size_t * transferred); You can do a plain read by passing probebuf=NULL, and you can do a plain probe by passing readbuf=NULL.
== Splitting File across multiple objects on different Servers == ==== Issue ==== * We likely need a way to split a large file across multiple objects on multiple servers. How do we handle updates safely across object boundaries? ==== Discussion ==== * do we need to support this within the storage system or can we force the client to handle it * does this really work in current file systems or do users really expect this to work? * Reddy will think more on this topic ==== Decision ==== * none
It works fine on some current file systems. Not all file systems over time have obtained all locks for v. large writes, but I do not know current status for Lustre, etc. GPFS probably does behave correctly.
== Can update_id decrease (or is it allowed to)? == ==== Issue ==== * Should we allow a decreasing update_id? ==== Discussion ==== * no one had a good reason to support this outside of reset ==== Decision ==== * do not allow reducing update_id, other than via reset and reset always goes to update_id = zero * May use same update_id on write * Zero is not a valid update_id to write
So clients have to coordinate if writing to multiple update_ids concurrently, or they will get an error? Surely I am misunderstanding. What does it mean to ³decrease² an update_id?
³Decreasing an update_id² is overwriting one or more records on a server with an update_id that is lower than those records already have. The reason for striking this was that Sirocco has problems doing that, because decreasing an update_id harms reconciling objects when there¹s a mismatch. The only way to keep that lower-update overwrite is extremely expensive and may not always work.
== Security Capabilities == ==== Issue ==== * how to integrate security into ASG API ==== Discussion ==== * (Dries) separate APIs, just add single capability argument to all other APIs * tabled discussion for now since it is not needed at this time
== Nesting of IDs in Sirocco == ==== Issue ==== * Sirocco allows generic nesting of ID spaces in the local store API, should this be extended to ASG? ==== Discussion ==== * any level of ID may store data * new API probe_child (which gives details about lower levels) * new API probe_data (which only gives data)
I¹d like to add that my argument against extending the ID space was because we wanted to have some address space available for the server to use for its own purposes, that the client shouldn¹t be able to access. I think that¹s the primary reason we didn¹t propose.
==== Decision ==== * don¹t propose to use this in ASG API at this time * this could be part of local store API
_______________________________________________ ASG mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/asg
_______________________________________________ ASG mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/asg
Hi Matthew, thanks for the response, more replies inline. — Rob On Mar 18, 2014, at 3:29 PM, Curry, Matthew Leon <[email protected]> wrote:
On 3/18/14, 11:17 AM, "Rob Ross" <[email protected]> wrote:
On Mar 17, 2014, at 5:10 PM, Harms, Kevin N. <[email protected]> wrote:
[ snip ]
== Read + Probe == ==== Issue ==== * race of contents changing between probe and read ==== Discussion ==== * versions may change between probe and read * solve with batch that enforces isolation ** (Phil) ASG was system API but batch is only for per-server which alters API usage * one solution with no isolation is probe+read in a single command * local transaction for a single server might be useful for this * Can this be used to make a consistent duplicate? (cloning) ==== Decision ==== * Update asg_read call to add Œprobe for record¹ parameters (may specify NULL if you don¹t want the probe data) * No change needed to support multiple probes, just always start at n+1 from previous returned call ==== Issue ==== * how to handle need to make multiple probe calls, if you can't return all results in one call. ==== Decision ==== * no change needed ** always probe (n+1)
Is this describing a mechanism for iterating through results of a probe call? If so, is some sort of cookie or something passed around with the call that enables the state to be serialized and not require tracking of clients? Assumption is that since the probe is relative to some update_id, results don¹t change (which would otherwise make things v. hard)?
This is describing how to iterate a probe when there are other servers inserting entries into the address space. The probe call does return the offset of the next record, but if another server were to insert something in the address space between the end of the last probe and the start of the next one, the server doing the probe would completely skip that insertion. It was decided that, if the server really cared about that use case, it could ignore the ³next offset² return value from the probe, and just probe from just beyond the end of the last interval it got.
Can you clarify what you mean by ³probe is relative to some update_id²?
Thanks. I read the header, so I am more informed now :). We should note somewhere what the instance_ids are for. We say that we do not “create a snapshot” with probe(s), and there is no update_id associated with a given container/object/fork for a user to use to reason about that set of changes they can perform while a probe is occurring (e.g., if I use larger update_ids while I write to this, I can probe on earlier ones and know I will get sane results). In PVFS we passed back a token that we used to avoid having to search to a specific ID, with the problem that changes to the thing we were iterating over (usually a directory) might result in duplicates in the output, etc. In the solution here we will need to search for the next ID each time, so every probe will incur a search (could be fast or slow, depending, obviously). If the use case is to iterate through all the things in a big set, then this could be high overhead as compared to a “give me back an iterator token”, but the iterator approach is flawed unless you have some way of creating an instance over which you can iterate and know you aren’t missing anything. Anyway, I don’t really have a suggestion on this front, just wanted to say that this is going to be a point of complexity for users.
==== Decision ==== * Switch to ³update_id² instead of version_id in ASG documentation
== Multiple Round-Trips for Probes and Reads == ==== Issue ==== * a large discontiguous file could result in requiring many probe+reads ==== Discussion ==== * (Matt) use probe+read, don¹t ask for probe results and zero-fill data (assume 1 byte records with zero value) * (Dries) have a method to request default record behavior (so for zero-fill, ask for default of 1-byte zero records) (requires storing this metadata with object/fork/etc) * (Dries) possible uses for security implementation ==== Decision ==== * none
Are we really proposing a fill capability on the object server side (and thus some semantic about filling records in order), or is this more about an ability to have the server create ³virtual² records that enable us to make more efficient requests?
This one deserves a little motivation. I was concerned about doing file I/O in cases where objects have random writes and/or lots of holes. Just by virtue of incrementing update_ids, you get into a situation where the probe data may very well be larger than the data you¹re reading. There were two solutions that were briefly discussed, and one that wasn¹t: 1. (Admittedly the weakest) Use zero fill on the server side to obviate the need for an accompanying probe. Dries suggested the alternative of allowing the client to set the default record. A flag would be used to dictate whether fill could be used, and the server could send whichever representation would be smaller. (see item 3 below) 2. Allow probes to accept a flag that said that update_ids aren¹t important, so a single record could be used for dense data with various record IDs and homogeneous record size. Better, but does not cover the case where there are lots of holes. 3. (The one I didn¹t bring up yet, because it¹s weird.) Allow a single buffer space to contain probe and read contents. Having separate buffers can cause the probe buffer to fill well before the read, giving us a lot of data that doesn¹t have a proper map for detecting holes or differently sized records, necessitating a reread. This ties into probe+read, which I¹ll talk about next.
Are we really happy with this probe+read approach? Seems like we¹re driving down a path to multiple operations when we only want one. What am I missing?
Dries (and Lee, privately, about a month ago) proposed a combined probe+read operation that does both with one call. Dries¹s idea of the signature is similar to:
int asg_probe_read ( asg_instance_t instance, asg_location_t location, asg_container_id_t container, asg_object_id_t object, asg_fork_id_t fork, asg_record_id_t start_record, asg_size_t recordcount, asg_flags_t flags, asg_version_t version_condition, void * readbuf, size_t readbufsize,
asg_record_info_t * probebuf, size_t probebufsize, asg_size_t * transferred);
You can do a plain read by passing probebuf=NULL, and you can do a plain probe by passing readbuf=NULL.
I assume this is really asg_probe_read_fork() and that what you get back is the results of the probe plus the data from the matches? It sounds like you worked out the use case of reading from a sparse fork that one would expect in the “I’m using an object to hold the data for a shard of a POSIX file, using single byte records”. I assume the above lets me say “give me all the data in this range” and I get back some data plus a list of what regions they correspond to, which should be a win 99% of the time (e.g., except in cases where someone wrote every other byte, or something similarly gnarly)? [ snip ]
== Can update_id decrease (or is it allowed to)? == ==== Issue ==== * Should we allow a decreasing update_id? ==== Discussion ==== * no one had a good reason to support this outside of reset ==== Decision ==== * do not allow reducing update_id, other than via reset and reset always goes to update_id = zero * May use same update_id on write * Zero is not a valid update_id to write
So clients have to coordinate if writing to multiple update_ids concurrently, or they will get an error? Surely I am misunderstanding. What does it mean to ³decrease² an update_id?
³Decreasing an update_id² is overwriting one or more records on a server with an update_id that is lower than those records already have. The reason for striking this was that Sirocco has problems doing that, because decreasing an update_id harms reconciling objects when there¹s a mismatch. The only way to keep that lower-update overwrite is extremely expensive and may not always work.
Ok, so specifically if someone writes to a record and specified an update_id lower than the current one, then they will get an error writing (or is it undefined?). Thus it is up to a user to coordinate updates at the record level. Makes sense. Thanks.
== Security Capabilities == ==== Issue ==== * how to integrate security into ASG API ==== Discussion ==== * (Dries) separate APIs, just add single capability argument to all other APIs * tabled discussion for now since it is not needed at this time
== Nesting of IDs in Sirocco == ==== Issue ==== * Sirocco allows generic nesting of ID spaces in the local store API, should this be extended to ASG? ==== Discussion ==== * any level of ID may store data * new API probe_child (which gives details about lower levels) * new API probe_data (which only gives data)
I¹d like to add that my argument against extending the ID space was because we wanted to have some address space available for the server to use for its own purposes, that the client shouldn¹t be able to access. I think that¹s the primary reason we didn¹t propose.
This can be done at the server level, as you say. Thanks for the detailed and thoughtful responses. Rob
Unreal. I totally forgot about this when preparing for my recent travels. Apologies. Thank you for the nice discussion. I did have a couple of related topics in my notes that I wanted to bring up. How might we discuss those? They were…. 1) Batches and transactions. We have an API and set of semantics for this. I am not entirely comfortable with our distributed transaction support and need thought and suggestions from the group. 2) Sirocco needs cache validation/invalidation calls. Triton should not need this but probably can accommodate the API without issue. Check with ANL on that! These were notes to self and item 2 needs some explanation. We think we need to be able to have a client tell a server that it should believe it is authoritative for a container+object and, similarly, that it should no longer believe it is if it already thinks it is. This is fallout from our free movement of data model. In essence we need a way for the client to promise one or more servers that it is making coherent and consistent updates so that the server is not required to search the system on each read request for the highest update ID. Similarly, the client needs to tell a server it has (or will immediately) make changes to an object that are (will be) not visible to the server. This raises an issue in that we need some way to restrict this idea of authority about the contents of an object to a coherence group made of cooperating clients. The "check with ANL" part is that I'm thinking the relevant Triton servers are always authoritative and can simply ignore the calls. Yes? If so, should the Triton server just say OK or return an error saying OK for the "become authoritative" call and return something like "not supported" for the invalidate call? Thanks and sorry again. --Lee On Mar 19, 2014, at 9:59 AM, Rob Ross wrote:
Hi Matthew, thanks for the response, more replies inline. — Rob
On Mar 18, 2014, at 3:29 PM, Curry, Matthew Leon <[email protected]> wrote:
On 3/18/14, 11:17 AM, "Rob Ross" <[email protected]> wrote:
On Mar 17, 2014, at 5:10 PM, Harms, Kevin N. <[email protected]> wrote:
[ snip ]
== Read + Probe == ==== Issue ==== * race of contents changing between probe and read ==== Discussion ==== * versions may change between probe and read * solve with batch that enforces isolation ** (Phil) ASG was system API but batch is only for per-server which alters API usage * one solution with no isolation is probe+read in a single command * local transaction for a single server might be useful for this * Can this be used to make a consistent duplicate? (cloning) ==== Decision ==== * Update asg_read call to add Œprobe for record¹ parameters (may specify NULL if you don¹t want the probe data) * No change needed to support multiple probes, just always start at n+1 from previous returned call ==== Issue ==== * how to handle need to make multiple probe calls, if you can't return all results in one call. ==== Decision ==== * no change needed ** always probe (n+1)
Is this describing a mechanism for iterating through results of a probe call? If so, is some sort of cookie or something passed around with the call that enables the state to be serialized and not require tracking of clients? Assumption is that since the probe is relative to some update_id, results don¹t change (which would otherwise make things v. hard)?
This is describing how to iterate a probe when there are other servers inserting entries into the address space. The probe call does return the offset of the next record, but if another server were to insert something in the address space between the end of the last probe and the start of the next one, the server doing the probe would completely skip that insertion. It was decided that, if the server really cared about that use case, it could ignore the ³next offset² return value from the probe, and just probe from just beyond the end of the last interval it got.
Can you clarify what you mean by ³probe is relative to some update_id²?
Thanks. I read the header, so I am more informed now :). We should note somewhere what the instance_ids are for.
We say that we do not “create a snapshot” with probe(s), and there is no update_id associated with a given container/object/fork for a user to use to reason about that set of changes they can perform while a probe is occurring (e.g., if I use larger update_ids while I write to this, I can probe on earlier ones and know I will get sane results).
In PVFS we passed back a token that we used to avoid having to search to a specific ID, with the problem that changes to the thing we were iterating over (usually a directory) might result in duplicates in the output, etc. In the solution here we will need to search for the next ID each time, so every probe will incur a search (could be fast or slow, depending, obviously). If the use case is to iterate through all the things in a big set, then this could be high overhead as compared to a “give me back an iterator token”, but the iterator approach is flawed unless you have some way of creating an instance over which you can iterate and know you aren’t missing anything.
Anyway, I don’t really have a suggestion on this front, just wanted to say that this is going to be a point of complexity for users.
==== Decision ==== * Switch to ³update_id² instead of version_id in ASG documentation
== Multiple Round-Trips for Probes and Reads == ==== Issue ==== * a large discontiguous file could result in requiring many probe+reads ==== Discussion ==== * (Matt) use probe+read, don¹t ask for probe results and zero-fill data (assume 1 byte records with zero value) * (Dries) have a method to request default record behavior (so for zero-fill, ask for default of 1-byte zero records) (requires storing this metadata with object/fork/etc) * (Dries) possible uses for security implementation ==== Decision ==== * none
Are we really proposing a fill capability on the object server side (and thus some semantic about filling records in order), or is this more about an ability to have the server create ³virtual² records that enable us to make more efficient requests?
This one deserves a little motivation. I was concerned about doing file I/O in cases where objects have random writes and/or lots of holes. Just by virtue of incrementing update_ids, you get into a situation where the probe data may very well be larger than the data you¹re reading. There were two solutions that were briefly discussed, and one that wasn¹t: 1. (Admittedly the weakest) Use zero fill on the server side to obviate the need for an accompanying probe. Dries suggested the alternative of allowing the client to set the default record. A flag would be used to dictate whether fill could be used, and the server could send whichever representation would be smaller. (see item 3 below) 2. Allow probes to accept a flag that said that update_ids aren¹t important, so a single record could be used for dense data with various record IDs and homogeneous record size. Better, but does not cover the case where there are lots of holes. 3. (The one I didn¹t bring up yet, because it¹s weird.) Allow a single buffer space to contain probe and read contents. Having separate buffers can cause the probe buffer to fill well before the read, giving us a lot of data that doesn¹t have a proper map for detecting holes or differently sized records, necessitating a reread. This ties into probe+read, which I¹ll talk about next.
Are we really happy with this probe+read approach? Seems like we¹re driving down a path to multiple operations when we only want one. What am I missing?
Dries (and Lee, privately, about a month ago) proposed a combined probe+read operation that does both with one call. Dries¹s idea of the signature is similar to:
int asg_probe_read ( asg_instance_t instance, asg_location_t location, asg_container_id_t container, asg_object_id_t object, asg_fork_id_t fork, asg_record_id_t start_record, asg_size_t recordcount, asg_flags_t flags, asg_version_t version_condition, void * readbuf, size_t readbufsize,
asg_record_info_t * probebuf, size_t probebufsize, asg_size_t * transferred);
You can do a plain read by passing probebuf=NULL, and you can do a plain probe by passing readbuf=NULL.
I assume this is really asg_probe_read_fork() and that what you get back is the results of the probe plus the data from the matches?
It sounds like you worked out the use case of reading from a sparse fork that one would expect in the “I’m using an object to hold the data for a shard of a POSIX file, using single byte records”. I assume the above lets me say “give me all the data in this range” and I get back some data plus a list of what regions they correspond to, which should be a win 99% of the time (e.g., except in cases where someone wrote every other byte, or something similarly gnarly)?
[ snip ]
== Can update_id decrease (or is it allowed to)? == ==== Issue ==== * Should we allow a decreasing update_id? ==== Discussion ==== * no one had a good reason to support this outside of reset ==== Decision ==== * do not allow reducing update_id, other than via reset and reset always goes to update_id = zero * May use same update_id on write * Zero is not a valid update_id to write
So clients have to coordinate if writing to multiple update_ids concurrently, or they will get an error? Surely I am misunderstanding. What does it mean to ³decrease² an update_id?
³Decreasing an update_id² is overwriting one or more records on a server with an update_id that is lower than those records already have. The reason for striking this was that Sirocco has problems doing that, because decreasing an update_id harms reconciling objects when there¹s a mismatch. The only way to keep that lower-update overwrite is extremely expensive and may not always work.
Ok, so specifically if someone writes to a record and specified an update_id lower than the current one, then they will get an error writing (or is it undefined?). Thus it is up to a user to coordinate updates at the record level. Makes sense. Thanks.
== Security Capabilities == ==== Issue ==== * how to integrate security into ASG API ==== Discussion ==== * (Dries) separate APIs, just add single capability argument to all other APIs * tabled discussion for now since it is not needed at this time
== Nesting of IDs in Sirocco == ==== Issue ==== * Sirocco allows generic nesting of ID spaces in the local store API, should this be extended to ASG? ==== Discussion ==== * any level of ID may store data * new API probe_child (which gives details about lower levels) * new API probe_data (which only gives data)
I¹d like to add that my argument against extending the ID space was because we wanted to have some address space available for the server to use for its own purposes, that the client shouldn¹t be able to access. I think that¹s the primary reason we didn¹t propose.
This can be done at the server level, as you say.
Thanks for the detailed and thoughtful responses.
Rob _______________________________________________ ASG mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/asg
I haven’t seen follow-up on this, so I will open a long-standing can of worms. I believe that (2) is a server API thing, and ASG (as a thing we are trying to share) is a service API and not a server API. Rob On Mar 27, 2014, at 11:34 AM, Ward, Lee <[email protected]> wrote:
Unreal. I totally forgot about this when preparing for my recent travels. Apologies.
Thank you for the nice discussion. I did have a couple of related topics in my notes that I wanted to bring up. How might we discuss those?
They were….
1) Batches and transactions. We have an API and set of semantics for this. I am not entirely comfortable with our distributed transaction support and need thought and suggestions from the group.
2) Sirocco needs cache validation/invalidation calls. Triton should not need this but probably can accommodate the API without issue. Check with ANL on that!
These were notes to self and item 2 needs some explanation. We think we need to be able to have a client tell a server that it should believe it is authoritative for a container+object and, similarly, that it should no longer believe it is if it already thinks it is. This is fallout from our free movement of data model. In essence we need a way for the client to promise one or more servers that it is making coherent and consistent updates so that the server is not required to search the system on each read request for the highest update ID. Similarly, the client needs to tell a server it has (or will immediately) make changes to an object that are (will be) not visible to the server. This raises an issue in that we need some way to restrict this idea of authority about the contents of an object to a coherence group made of cooperating clients.
The "check with ANL" part is that I'm thinking the relevant Triton servers are always authoritative and can simply ignore the calls. Yes? If so, should the Triton server just say OK or return an error saying OK for the "become authoritative" call and return something like "not supported" for the invalidate call?
Thanks and sorry again.
--Lee
On Mar 19, 2014, at 9:59 AM, Rob Ross wrote:
Hi Matthew, thanks for the response, more replies inline. — Rob
On Mar 18, 2014, at 3:29 PM, Curry, Matthew Leon <[email protected]> wrote:
On 3/18/14, 11:17 AM, "Rob Ross" <[email protected]> wrote:
On Mar 17, 2014, at 5:10 PM, Harms, Kevin N. <[email protected]> wrote:
[ snip ]
== Read + Probe == ==== Issue ==== * race of contents changing between probe and read ==== Discussion ==== * versions may change between probe and read * solve with batch that enforces isolation ** (Phil) ASG was system API but batch is only for per-server which alters API usage * one solution with no isolation is probe+read in a single command * local transaction for a single server might be useful for this * Can this be used to make a consistent duplicate? (cloning) ==== Decision ==== * Update asg_read call to add Œprobe for record¹ parameters (may specify NULL if you don¹t want the probe data) * No change needed to support multiple probes, just always start at n+1 from previous returned call ==== Issue ==== * how to handle need to make multiple probe calls, if you can't return all results in one call. ==== Decision ==== * no change needed ** always probe (n+1)
Is this describing a mechanism for iterating through results of a probe call? If so, is some sort of cookie or something passed around with the call that enables the state to be serialized and not require tracking of clients? Assumption is that since the probe is relative to some update_id, results don¹t change (which would otherwise make things v. hard)?
This is describing how to iterate a probe when there are other servers inserting entries into the address space. The probe call does return the offset of the next record, but if another server were to insert something in the address space between the end of the last probe and the start of the next one, the server doing the probe would completely skip that insertion. It was decided that, if the server really cared about that use case, it could ignore the ³next offset² return value from the probe, and just probe from just beyond the end of the last interval it got.
Can you clarify what you mean by ³probe is relative to some update_id²?
Thanks. I read the header, so I am more informed now :). We should note somewhere what the instance_ids are for.
We say that we do not “create a snapshot” with probe(s), and there is no update_id associated with a given container/object/fork for a user to use to reason about that set of changes they can perform while a probe is occurring (e.g., if I use larger update_ids while I write to this, I can probe on earlier ones and know I will get sane results).
In PVFS we passed back a token that we used to avoid having to search to a specific ID, with the problem that changes to the thing we were iterating over (usually a directory) might result in duplicates in the output, etc. In the solution here we will need to search for the next ID each time, so every probe will incur a search (could be fast or slow, depending, obviously). If the use case is to iterate through all the things in a big set, then this could be high overhead as compared to a “give me back an iterator token”, but the iterator approach is flawed unless you have some way of creating an instance over which you can iterate and know you aren’t missing anything.
Anyway, I don’t really have a suggestion on this front, just wanted to say that this is going to be a point of complexity for users.
==== Decision ==== * Switch to ³update_id² instead of version_id in ASG documentation
== Multiple Round-Trips for Probes and Reads == ==== Issue ==== * a large discontiguous file could result in requiring many probe+reads ==== Discussion ==== * (Matt) use probe+read, don¹t ask for probe results and zero-fill data (assume 1 byte records with zero value) * (Dries) have a method to request default record behavior (so for zero-fill, ask for default of 1-byte zero records) (requires storing this metadata with object/fork/etc) * (Dries) possible uses for security implementation ==== Decision ==== * none
Are we really proposing a fill capability on the object server side (and thus some semantic about filling records in order), or is this more about an ability to have the server create ³virtual² records that enable us to make more efficient requests?
This one deserves a little motivation. I was concerned about doing file I/O in cases where objects have random writes and/or lots of holes. Just by virtue of incrementing update_ids, you get into a situation where the probe data may very well be larger than the data you¹re reading. There were two solutions that were briefly discussed, and one that wasn¹t: 1. (Admittedly the weakest) Use zero fill on the server side to obviate the need for an accompanying probe. Dries suggested the alternative of allowing the client to set the default record. A flag would be used to dictate whether fill could be used, and the server could send whichever representation would be smaller. (see item 3 below) 2. Allow probes to accept a flag that said that update_ids aren¹t important, so a single record could be used for dense data with various record IDs and homogeneous record size. Better, but does not cover the case where there are lots of holes. 3. (The one I didn¹t bring up yet, because it¹s weird.) Allow a single buffer space to contain probe and read contents. Having separate buffers can cause the probe buffer to fill well before the read, giving us a lot of data that doesn¹t have a proper map for detecting holes or differently sized records, necessitating a reread. This ties into probe+read, which I¹ll talk about next.
Are we really happy with this probe+read approach? Seems like we¹re driving down a path to multiple operations when we only want one. What am I missing?
Dries (and Lee, privately, about a month ago) proposed a combined probe+read operation that does both with one call. Dries¹s idea of the signature is similar to:
int asg_probe_read ( asg_instance_t instance, asg_location_t location, asg_container_id_t container, asg_object_id_t object, asg_fork_id_t fork, asg_record_id_t start_record, asg_size_t recordcount, asg_flags_t flags, asg_version_t version_condition, void * readbuf, size_t readbufsize,
asg_record_info_t * probebuf, size_t probebufsize, asg_size_t * transferred);
You can do a plain read by passing probebuf=NULL, and you can do a plain probe by passing readbuf=NULL.
I assume this is really asg_probe_read_fork() and that what you get back is the results of the probe plus the data from the matches?
It sounds like you worked out the use case of reading from a sparse fork that one would expect in the “I’m using an object to hold the data for a shard of a POSIX file, using single byte records”. I assume the above lets me say “give me all the data in this range” and I get back some data plus a list of what regions they correspond to, which should be a win 99% of the time (e.g., except in cases where someone wrote every other byte, or something similarly gnarly)?
[ snip ]
== Can update_id decrease (or is it allowed to)? == ==== Issue ==== * Should we allow a decreasing update_id? ==== Discussion ==== * no one had a good reason to support this outside of reset ==== Decision ==== * do not allow reducing update_id, other than via reset and reset always goes to update_id = zero * May use same update_id on write * Zero is not a valid update_id to write
So clients have to coordinate if writing to multiple update_ids concurrently, or they will get an error? Surely I am misunderstanding. What does it mean to ³decrease² an update_id?
³Decreasing an update_id² is overwriting one or more records on a server with an update_id that is lower than those records already have. The reason for striking this was that Sirocco has problems doing that, because decreasing an update_id harms reconciling objects when there¹s a mismatch. The only way to keep that lower-update overwrite is extremely expensive and may not always work.
Ok, so specifically if someone writes to a record and specified an update_id lower than the current one, then they will get an error writing (or is it undefined?). Thus it is up to a user to coordinate updates at the record level. Makes sense. Thanks.
== Security Capabilities == ==== Issue ==== * how to integrate security into ASG API ==== Discussion ==== * (Dries) separate APIs, just add single capability argument to all other APIs * tabled discussion for now since it is not needed at this time
== Nesting of IDs in Sirocco == ==== Issue ==== * Sirocco allows generic nesting of ID spaces in the local store API, should this be extended to ASG? ==== Discussion ==== * any level of ID may store data * new API probe_child (which gives details about lower levels) * new API probe_data (which only gives data)
I¹d like to add that my argument against extending the ID space was because we wanted to have some address space available for the server to use for its own purposes, that the client shouldn¹t be able to access. I think that¹s the primary reason we didn¹t propose.
This can be done at the server level, as you say.
Thanks for the detailed and thoughtful responses.
Rob _______________________________________________ ASG mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/asg
Dries, I will join the call. Matthew On 3/5/14, 5:34 PM, "Dries Kimpe" <[email protected]> wrote:
Hi,
I'd like to reserve the usual time on Mon 3/16 to discuss our shared client API. We have about one quarter left to adopt a shared client API, and while we have a good start (the one we designed a long time ago) we all found some things that are not as nice or efficient as they could be. Some things just need to be clarified.
Some examples: - the probe+read race condition/inefficiency. - can update id/version decrease? - is it valid to write a 0-version (or can only reset do that)? ...
So, I'd like to dedicate 3/16, 3-4PM CST (usual concall number) to review the API.
This will mainly be relevant for the Sirocco/Triton teams (as well as TAMU, who can share their experiences implementing the posix client on top of this API).
For preparation, I would ask that everybody tries to prepare a list of issues we can go over in the call.
Please confirm if you can make the call (which should be no problem, since if I recall correctly, we were going to keep this time open every week even if the call was only once a month).
Thanks, Dries
_______________________________________________ ASG mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/asg
participants (5)
-
Curry, Matthew Leon -
Dries Kimpe -
Harms, Kevin N. -
Rob Ross -
Ward, Lee