Can I move a static mutex around in memory?
Hi, I would like to understand statically initialized mutex and condition variables better, in particular: can I move them around in memory (assuming all the ULTs in my code have a consistent way of locating them)? This would come up in C++ containers, for instance: std::vector<ABT_mutex_memory> If I increase the size of the vector, reallocation may take place, and the existing mutexes will be moved somewhere else in memory. Assuming all the ULTs are accessing them by their index in the vector rather than by an ABT_mutex opaque pointer, then this should be fine as long as I am allowed to move ABT_mutex_memory around (regardless of their state). Is that the case? Same question for condition variables? Thanks, Matthieu
Hi Matthieu, Thanks for the question! ABT_mutex -> Yes, it's internally just a pointer (more specifically, a handle), so you can copy it. ABT_mutex_memory -> No, it saves it states in this memory (directly used as ABTI_mutex), so if you copy it, you basically corrupt the ABT_mutex structure. https://github.com/pmodels/argobots/blob/main/src/include/abt.h.in#L1114-L11... The same thing goes to ABT_cond_memory: do not move ABT_cond_memory around. // Example (THE FOLLOWING IS FORBIDDEN) ABT_mutex_memory m1 = ABT_MUTEX_INITIALIZER; ABT_mutex_lock(ABT_MUTEX_MEMORY_GET_HANDLE(&m1)); ABT_mutex_memory m2; memcpy(&m2, &m1, sizeof(ABT_mutex_memory)); // m2 is locked since m1 is locked. The mutex "entity" is copied, so even if you unlock m2, m1 is not unlocked. // It can cause something even worse since this mutex entity may store the waiters or even a pointer to its own memory. // Do not use ABT_mutex_memory in this way. ABT_mutex_lock(ABT_MUTEX_MEMORY_GET_HANDLE(&m2)); In your case, reallocation on resizing std::vector can break the mutex data (even if ABT_mutex_memory is somehow moved "uniquely" ). Please dynamically allocate each ABT_mutex_memory individually (std::vector<ABT_mutex_memory *>). Thanks, Shintaro ________________________________ From: Dorier, Matthieu via discuss <[email protected]> Sent: Tuesday, June 15, 2021 7:04 AM To: [email protected] <[email protected]> Cc: Dorier, Matthieu <[email protected]> Subject: [argobots-discuss] Can I move a static mutex around in memory? Hi, I would like to understand statically initialized mutex and condition variables better, in particular: can I move them around in memory (assuming all the ULTs in my code have a consistent way of locating them)? This would come up in C++ containers, for instance: std::vector<ABT_mutex_memory> If I increase the size of the vector, reallocation may take place, and the existing mutexes will be moved somewhere else in memory. Assuming all the ULTs are accessing them by their index in the vector rather than by an ABT_mutex opaque pointer, then this should be fine as long as I am allowed to move ABT_mutex_memory around (regardless of their state). Is that the case? Same question for condition variables? Thanks, Matthieu
Hi Shintaro, Thank you for your answer. In the example you show, you say “even if you unlock m2, m1 is not unlocked”. That said, if no ULT cares about m1 anymore, and all the ULTs know they should use m2, then why shouldn’t it be fine? Maybe other objects in Argobots (such as an ES) are keeping pointers to mutexes, and these pointers would be invalidated? Thanks, Matthieu From: "Iwasaki, Shintaro" <[email protected]> Date: Tuesday, 15 June 2021 at 14:57 To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Cc: "Dorier, Matthieu" <[email protected]> Subject: Re: Can I move a static mutex around in memory? Hi Matthieu, Thanks for the question! ABT_mutex -> Yes, it's internally just a pointer (more specifically, a handle), so you can copy it. ABT_mutex_memory -> No, it saves it states in this memory (directly used as ABTI_mutex), so if you copy it, you basically corrupt the ABT_mutex structure. https://github.com/pmodels/argobots/blob/main/src/include/abt.h.in#L1114-L11... The same thing goes to ABT_cond_memory: do not move ABT_cond_memory around. // Example (THE FOLLOWING IS FORBIDDEN) ABT_mutex_memory m1 = ABT_MUTEX_INITIALIZER; ABT_mutex_lock(ABT_MUTEX_MEMORY_GET_HANDLE(&m1)); ABT_mutex_memory m2; memcpy(&m2, &m1, sizeof(ABT_mutex_memory)); // m2 is locked since m1 is locked. The mutex "entity" is copied, so even if you unlock m2, m1 is not unlocked. // It can cause something even worse since this mutex entity may store the waiters or even a pointer to its own memory. // Do not use ABT_mutex_memory in this way. ABT_mutex_lock(ABT_MUTEX_MEMORY_GET_HANDLE(&m2)); In your case, reallocation on resizing std::vector can break the mutex data (even if ABT_mutex_memory is somehow moved "uniquely" ). Please dynamically allocate each ABT_mutex_memory individually (std::vector<ABT_mutex_memory *>). Thanks, Shintaro ________________________________ From: Dorier, Matthieu via discuss <[email protected]> Sent: Tuesday, June 15, 2021 7:04 AM To: [email protected] <[email protected]> Cc: Dorier, Matthieu <[email protected]> Subject: [argobots-discuss] Can I move a static mutex around in memory? Hi, I would like to understand statically initialized mutex and condition variables better, in particular: can I move them around in memory (assuming all the ULTs in my code have a consistent way of locating them)? This would come up in C++ containers, for instance: std::vector<ABT_mutex_memory> If I increase the size of the vector, reallocation may take place, and the existing mutexes will be moved somewhere else in memory. Assuming all the ULTs are accessing them by their index in the vector rather than by an ABT_mutex opaque pointer, then this should be fine as long as I am allowed to move ABT_mutex_memory around (regardless of their state). Is that the case? Same question for condition variables? Thanks, Matthieu
Hi Matthieu, Thanks. In the example you show, you say “even if you unlock m2, m1 is not unlocked”. That said, if no ULT cares about m1 anymore, and all the ULTs know they should use m2, then why shouldn’t it be fine? It depends on the implementation. In my understanding, it is fine in the current Argobots implementation, but it can cause an issue in the future update. So far, I do not plan to ensure this property. For example, a possible Argobots implementation that can break this m1->m2 scenario is using a doubly circular linked list to manage waiters while one of the elements (maybe a head/tail marker of this linked list) is allocated within ABT_mutex_memory (= the internal mutex entity). If you memory-copy m1 to m2, a waiter of this mutex points to m1's memory address though m1 is no longer valid. Even if m1 is unlocked, I am not fully sure if copying m1 to m2 and discarding m1 is okay. In other words, locked-and-then-unlocked ABT_mutex_memory's state can be different from clean ABT_mutex_memory's state just after ABT_MUTEX_INITIALIZER, though I cannot come up with a good example that can cause such. Note that if you can guarantee the following, you might want to reinitialize std::vector<ABT_mutex_memory> every time, which should be robust and not affected by the future update. 1. When std::vector<ABT_mutex_memory> is resized, none of them are "locked" (implied by "no ULT cares about m1 anymore") 2. When std::vector<ABT_mutex_memory> is resized, there is no access to the old one: all ULTs can access new ABT_mutex without any data race (implied by "all the ULTs know they should use m2"). The code should look like this: ``` std::vector<ABT_mutex_memory> mutex_mems; mutex_mems.resize(16); for (int i = 0; i < mutex_mems.size(); i++) { // Clean up mutex_mems. ABT_mutex_free is not needed to release statically allocated ABT_mutex. const ABT_mutex_memory mutex_mem_init = ABT_MUTEX_INITIALIZER; memcpy(&mutex_mems[i], &mutex_mem_init, sizeof(ABT_mutex_memory)); } ``` If this memcpy() looks awkward, Argobots can provide a single-line macro. I would appreciate any suggestions! Thanks, Shintaro ________________________________ From: Dorier, Matthieu <[email protected]> Sent: Tuesday, June 15, 2021 10:09 AM To: Iwasaki, Shintaro <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]> Subject: Re: Can I move a static mutex around in memory? Hi Shintaro, Thank you for your answer. In the example you show, you say “even if you unlock m2, m1 is not unlocked”. That said, if no ULT cares about m1 anymore, and all the ULTs know they should use m2, then why shouldn’t it be fine? Maybe other objects in Argobots (such as an ES) are keeping pointers to mutexes, and these pointers would be invalidated? Thanks, Matthieu From: "Iwasaki, Shintaro" <[email protected]> Date: Tuesday, 15 June 2021 at 14:57 To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Cc: "Dorier, Matthieu" <[email protected]> Subject: Re: Can I move a static mutex around in memory? Hi Matthieu, Thanks for the question! ABT_mutex -> Yes, it's internally just a pointer (more specifically, a handle), so you can copy it. ABT_mutex_memory -> No, it saves it states in this memory (directly used as ABTI_mutex), so if you copy it, you basically corrupt the ABT_mutex structure. https://github.com/pmodels/argobots/blob/main/src/include/abt.h.in#L1114-L11... The same thing goes to ABT_cond_memory: do not move ABT_cond_memory around. // Example (THE FOLLOWING IS FORBIDDEN) ABT_mutex_memory m1 = ABT_MUTEX_INITIALIZER; ABT_mutex_lock(ABT_MUTEX_MEMORY_GET_HANDLE(&m1)); ABT_mutex_memory m2; memcpy(&m2, &m1, sizeof(ABT_mutex_memory)); // m2 is locked since m1 is locked. The mutex "entity" is copied, so even if you unlock m2, m1 is not unlocked. // It can cause something even worse since this mutex entity may store the waiters or even a pointer to its own memory. // Do not use ABT_mutex_memory in this way. ABT_mutex_lock(ABT_MUTEX_MEMORY_GET_HANDLE(&m2)); In your case, reallocation on resizing std::vector can break the mutex data (even if ABT_mutex_memory is somehow moved "uniquely" ). Please dynamically allocate each ABT_mutex_memory individually (std::vector<ABT_mutex_memory *>). Thanks, Shintaro ________________________________ From: Dorier, Matthieu via discuss <[email protected]> Sent: Tuesday, June 15, 2021 7:04 AM To: [email protected] <[email protected]> Cc: Dorier, Matthieu <[email protected]> Subject: [argobots-discuss] Can I move a static mutex around in memory? Hi, I would like to understand statically initialized mutex and condition variables better, in particular: can I move them around in memory (assuming all the ULTs in my code have a consistent way of locating them)? This would come up in C++ containers, for instance: std::vector<ABT_mutex_memory> If I increase the size of the vector, reallocation may take place, and the existing mutexes will be moved somewhere else in memory. Assuming all the ULTs are accessing them by their index in the vector rather than by an ABT_mutex opaque pointer, then this should be fine as long as I am allowed to move ABT_mutex_memory around (regardless of their state). Is that the case? Same question for condition variables? Thanks, Matthieu
Ok thanks, so it seems I should not count too much on those mutexes being movable. I was asking because in Thallium I have a C++ mutex class wrapping an ABT_mutex (https://github.com/mochi-hpc/mochi-thallium/blob/main/include/thallium/mutex...). For now its constructor uses ABT_mutex_create and its destructor ABT_mutex_free. It has move operations but no copy operations. If the ABT_mutex_memory had been movable, I could have replaced the ABT_mutex in this class with an ABT_mutex_memory. By the way, do static mutexes have an advantage beyond avoiding an allocation on the heap (which generally is not in the critical path when it comes to mutexes) and shortening code a bit? (if those are the only advantages, then there is not much reason for me to update my mutex class). Thanks, Matthieu From: "Iwasaki, Shintaro" <[email protected]> Date: Tuesday, 15 June 2021 at 16:46 To: "Dorier, Matthieu" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: Can I move a static mutex around in memory? Hi Matthieu, Thanks. In the example you show, you say “even if you unlock m2, m1 is not unlocked”. That said, if no ULT cares about m1 anymore, and all the ULTs know they should use m2, then why shouldn’t it be fine? It depends on the implementation. In my understanding, it is fine in the current Argobots implementation, but it can cause an issue in the future update. So far, I do not plan to ensure this property. For example, a possible Argobots implementation that can break this m1->m2 scenario is using a doubly circular linked list to manage waiters while one of the elements (maybe a head/tail marker of this linked list) is allocated within ABT_mutex_memory (= the internal mutex entity). If you memory-copy m1 to m2, a waiter of this mutex points to m1's memory address though m1 is no longer valid. Even if m1 is unlocked, I am not fully sure if copying m1 to m2 and discarding m1 is okay. In other words, locked-and-then-unlocked ABT_mutex_memory's state can be different from clean ABT_mutex_memory's state just after ABT_MUTEX_INITIALIZER, though I cannot come up with a good example that can cause such. Note that if you can guarantee the following, you might want to reinitialize std::vector<ABT_mutex_memory> every time, which should be robust and not affected by the future update. 1. When std::vector<ABT_mutex_memory> is resized, none of them are "locked" (implied by "no ULT cares about m1 anymore") 2. When std::vector<ABT_mutex_memory> is resized, there is no access to the old one: all ULTs can access new ABT_mutex without any data race (implied by "all the ULTs know they should use m2"). The code should look like this: ``` std::vector<ABT_mutex_memory> mutex_mems; mutex_mems.resize(16); for (int i = 0; i < mutex_mems.size(); i++) { // Clean up mutex_mems. ABT_mutex_free is not needed to release statically allocated ABT_mutex. const ABT_mutex_memory mutex_mem_init = ABT_MUTEX_INITIALIZER; memcpy(&mutex_mems[i], &mutex_mem_init, sizeof(ABT_mutex_memory)); } ``` If this memcpy() looks awkward, Argobots can provide a single-line macro. I would appreciate any suggestions! Thanks, Shintaro ________________________________ From: Dorier, Matthieu <[email protected]> Sent: Tuesday, June 15, 2021 10:09 AM To: Iwasaki, Shintaro <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]> Subject: Re: Can I move a static mutex around in memory? Hi Shintaro, Thank you for your answer. In the example you show, you say “even if you unlock m2, m1 is not unlocked”. That said, if no ULT cares about m1 anymore, and all the ULTs know they should use m2, then why shouldn’t it be fine? Maybe other objects in Argobots (such as an ES) are keeping pointers to mutexes, and these pointers would be invalidated? Thanks, Matthieu From: "Iwasaki, Shintaro" <[email protected]> Date: Tuesday, 15 June 2021 at 14:57 To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Cc: "Dorier, Matthieu" <[email protected]> Subject: Re: Can I move a static mutex around in memory? Hi Matthieu, Thanks for the question! ABT_mutex -> Yes, it's internally just a pointer (more specifically, a handle), so you can copy it. ABT_mutex_memory -> No, it saves it states in this memory (directly used as ABTI_mutex), so if you copy it, you basically corrupt the ABT_mutex structure. https://github.com/pmodels/argobots/blob/main/src/include/abt.h.in#L1114-L11... The same thing goes to ABT_cond_memory: do not move ABT_cond_memory around. // Example (THE FOLLOWING IS FORBIDDEN) ABT_mutex_memory m1 = ABT_MUTEX_INITIALIZER; ABT_mutex_lock(ABT_MUTEX_MEMORY_GET_HANDLE(&m1)); ABT_mutex_memory m2; memcpy(&m2, &m1, sizeof(ABT_mutex_memory)); // m2 is locked since m1 is locked. The mutex "entity" is copied, so even if you unlock m2, m1 is not unlocked. // It can cause something even worse since this mutex entity may store the waiters or even a pointer to its own memory. // Do not use ABT_mutex_memory in this way. ABT_mutex_lock(ABT_MUTEX_MEMORY_GET_HANDLE(&m2)); In your case, reallocation on resizing std::vector can break the mutex data (even if ABT_mutex_memory is somehow moved "uniquely" ). Please dynamically allocate each ABT_mutex_memory individually (std::vector<ABT_mutex_memory *>). Thanks, Shintaro ________________________________ From: Dorier, Matthieu via discuss <[email protected]> Sent: Tuesday, June 15, 2021 7:04 AM To: [email protected] <[email protected]> Cc: Dorier, Matthieu <[email protected]> Subject: [argobots-discuss] Can I move a static mutex around in memory? Hi, I would like to understand statically initialized mutex and condition variables better, in particular: can I move them around in memory (assuming all the ULTs in my code have a consistent way of locating them)? This would come up in C++ containers, for instance: std::vector<ABT_mutex_memory> If I increase the size of the vector, reallocation may take place, and the existing mutexes will be moved somewhere else in memory. Assuming all the ULTs are accessing them by their index in the vector rather than by an ABT_mutex opaque pointer, then this should be fine as long as I am allowed to move ABT_mutex_memory around (regardless of their state). Is that the case? Same question for condition variables? Thanks, Matthieu
Hi Matthieu, Unfortunately, currently ABT_{mutex,cond}_memory is not movable as you wrote. By the way, do static mutexes have an advantage beyond avoiding an allocation on the heap (which generally is not in the critical path when it comes to mutexes) and shortening code a bit? The primary advantage of static mutex/cond is that it can be used to protect global data outside ABT_init() - ABT_finalize() scope. Now some ABT_mutex_xxx() and ABT_cond_xxx() functions can be used without Argobots initialization, so the user can easily integrate Argobots for a runtime that does not have explicit initialization/finalization (e.g., MPI has such a scope (MPI_Init()/MPI_Finalize()) but OpenMP does not have such) but needs to protect something from parallel ULT/Pthreads accesses. See https://github.com/pmodels/argobots/pull/299. There's no performance/feature difference between dynamic and static mutexes except for the create/free cost. Thanks, Shintaro ________________________________ From: Dorier, Matthieu <[email protected]> Sent: Tuesday, June 15, 2021 10:57 AM To: Iwasaki, Shintaro <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]> Subject: Re: Can I move a static mutex around in memory? Ok thanks, so it seems I should not count too much on those mutexes being movable. I was asking because in Thallium I have a C++ mutex class wrapping an ABT_mutex (https://github.com/mochi-hpc/mochi-thallium/blob/main/include/thallium/mutex...). For now its constructor uses ABT_mutex_create and its destructor ABT_mutex_free. It has move operations but no copy operations. If the ABT_mutex_memory had been movable, I could have replaced the ABT_mutex in this class with an ABT_mutex_memory. By the way, do static mutexes have an advantage beyond avoiding an allocation on the heap (which generally is not in the critical path when it comes to mutexes) and shortening code a bit? (if those are the only advantages, then there is not much reason for me to update my mutex class). Thanks, Matthieu From: "Iwasaki, Shintaro" <[email protected]> Date: Tuesday, 15 June 2021 at 16:46 To: "Dorier, Matthieu" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: Can I move a static mutex around in memory? Hi Matthieu, Thanks. In the example you show, you say “even if you unlock m2, m1 is not unlocked”. That said, if no ULT cares about m1 anymore, and all the ULTs know they should use m2, then why shouldn’t it be fine? It depends on the implementation. In my understanding, it is fine in the current Argobots implementation, but it can cause an issue in the future update. So far, I do not plan to ensure this property. For example, a possible Argobots implementation that can break this m1->m2 scenario is using a doubly circular linked list to manage waiters while one of the elements (maybe a head/tail marker of this linked list) is allocated within ABT_mutex_memory (= the internal mutex entity). If you memory-copy m1 to m2, a waiter of this mutex points to m1's memory address though m1 is no longer valid. Even if m1 is unlocked, I am not fully sure if copying m1 to m2 and discarding m1 is okay. In other words, locked-and-then-unlocked ABT_mutex_memory's state can be different from clean ABT_mutex_memory's state just after ABT_MUTEX_INITIALIZER, though I cannot come up with a good example that can cause such. Note that if you can guarantee the following, you might want to reinitialize std::vector<ABT_mutex_memory> every time, which should be robust and not affected by the future update. 1. When std::vector<ABT_mutex_memory> is resized, none of them are "locked" (implied by "no ULT cares about m1 anymore") 2. When std::vector<ABT_mutex_memory> is resized, there is no access to the old one: all ULTs can access new ABT_mutex without any data race (implied by "all the ULTs know they should use m2"). The code should look like this: ``` std::vector<ABT_mutex_memory> mutex_mems; mutex_mems.resize(16); for (int i = 0; i < mutex_mems.size(); i++) { // Clean up mutex_mems. ABT_mutex_free is not needed to release statically allocated ABT_mutex. const ABT_mutex_memory mutex_mem_init = ABT_MUTEX_INITIALIZER; memcpy(&mutex_mems[i], &mutex_mem_init, sizeof(ABT_mutex_memory)); } ``` If this memcpy() looks awkward, Argobots can provide a single-line macro. I would appreciate any suggestions! Thanks, Shintaro ________________________________ From: Dorier, Matthieu <[email protected]> Sent: Tuesday, June 15, 2021 10:09 AM To: Iwasaki, Shintaro <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]> Subject: Re: Can I move a static mutex around in memory? Hi Shintaro, Thank you for your answer. In the example you show, you say “even if you unlock m2, m1 is not unlocked”. That said, if no ULT cares about m1 anymore, and all the ULTs know they should use m2, then why shouldn’t it be fine? Maybe other objects in Argobots (such as an ES) are keeping pointers to mutexes, and these pointers would be invalidated? Thanks, Matthieu From: "Iwasaki, Shintaro" <[email protected]> Date: Tuesday, 15 June 2021 at 14:57 To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Cc: "Dorier, Matthieu" <[email protected]> Subject: Re: Can I move a static mutex around in memory? Hi Matthieu, Thanks for the question! ABT_mutex -> Yes, it's internally just a pointer (more specifically, a handle), so you can copy it. ABT_mutex_memory -> No, it saves it states in this memory (directly used as ABTI_mutex), so if you copy it, you basically corrupt the ABT_mutex structure. https://github.com/pmodels/argobots/blob/main/src/include/abt.h.in#L1114-L11... The same thing goes to ABT_cond_memory: do not move ABT_cond_memory around. // Example (THE FOLLOWING IS FORBIDDEN) ABT_mutex_memory m1 = ABT_MUTEX_INITIALIZER; ABT_mutex_lock(ABT_MUTEX_MEMORY_GET_HANDLE(&m1)); ABT_mutex_memory m2; memcpy(&m2, &m1, sizeof(ABT_mutex_memory)); // m2 is locked since m1 is locked. The mutex "entity" is copied, so even if you unlock m2, m1 is not unlocked. // It can cause something even worse since this mutex entity may store the waiters or even a pointer to its own memory. // Do not use ABT_mutex_memory in this way. ABT_mutex_lock(ABT_MUTEX_MEMORY_GET_HANDLE(&m2)); In your case, reallocation on resizing std::vector can break the mutex data (even if ABT_mutex_memory is somehow moved "uniquely" ). Please dynamically allocate each ABT_mutex_memory individually (std::vector<ABT_mutex_memory *>). Thanks, Shintaro ________________________________ From: Dorier, Matthieu via discuss <[email protected]> Sent: Tuesday, June 15, 2021 7:04 AM To: [email protected] <[email protected]> Cc: Dorier, Matthieu <[email protected]> Subject: [argobots-discuss] Can I move a static mutex around in memory? Hi, I would like to understand statically initialized mutex and condition variables better, in particular: can I move them around in memory (assuming all the ULTs in my code have a consistent way of locating them)? This would come up in C++ containers, for instance: std::vector<ABT_mutex_memory> If I increase the size of the vector, reallocation may take place, and the existing mutexes will be moved somewhere else in memory. Assuming all the ULTs are accessing them by their index in the vector rather than by an ABT_mutex opaque pointer, then this should be fine as long as I am allowed to move ABT_mutex_memory around (regardless of their state). Is that the case? Same question for condition variables? Thanks, Matthieu
participants (2)
-
Dorier, Matthieu -
Iwasaki, Shintaro