- discuss - mailman3.cels.anl.gov

ABT segfault in MacOS
by Jean Luca Bez 25 Jun '21

25 Jun '21

Hi, I compiled the last git version of Argobots in MacOS, ran the tests and they all passed. However, when running an application I am getting some strange segfault that appears to come from ABT. Does anyone have faced similar issues? *** Process received signal *** Signal: Segmentation fault: 11 (11) Signal code: (0) Failing at address: 0x0 [ 0] 0 libsystem_platform.dylib 0x00007fff20428d7d _sigtramp + 29 [ 1] 0 ??? 0x0000000000000000 0x0 + 0 [ 2] 0 libabt.1.dylib 0x0000000105bdbdc0 ABT_thread_create + 128 [ 3] 0 libh5async.dylib 0x00000001064bde1f push_task_to_abt_pool + 559 [ 4] 0 libh5async.dylib 0x00000001064e6a02 async_group_create + 1890 [ 5] 0 libh5async.dylib 0x00000001064c4061 H5VL_async_group_create + 321 [ 6] 0 libhdf5.1000.dylib 0x0000000105f6f794 H5VL__group_create + 180 [ 7] 0 libhdf5.1000.dylib 0x0000000105f6f569 H5VL_group_create + 217 [ 8] 0 libhdf5.1000.dylib 0x0000000105d48a04 H5G__create_api_common + 660 [ 9] 0 libhdf5.1000.dylib 0x0000000105d485f5 H5Gcreate2 + 325 [10] 0 async_test_parallel.exe 0x0000000105bbcb43 main + 739 [11] 0 libdyld.dylib 0x00007fff203fef5d start + 1 [12] 0 ??? 0x0000000000000001 0x0 + 1 Thank you, Jean Luca

2 3

Attribution of ULT ids
by Dorier, Matthieu 22 Jun '21

22 Jun '21

Hi, Is the ABT_unit_id associated to threads and tasks a number that necessarily increases as new threads/tasks are created? I’m asking because I would like to implement a custom pool that prioritizes running older ULTs over newer ones when possible, and one way of knowing which of multiple ULTs is older could be to look at the ABT_unit_id and prioritize ULTs with the smallest id. Thanks, Matthieu

2 2

Can I move a static mutex around in memory?
by Dorier, Matthieu 15 Jun '21

15 Jun '21

Hi, I would like to understand statically initialized mutex and condition variables better, in particular: can I move them around in memory (assuming all the ULTs in my code have a consistent way of locating them)? This would come up in C++ containers, for instance: std::vector<ABT_mutex_memory> If I increase the size of the vector, reallocation may take place, and the existing mutexes will be moved somewhere else in memory. Assuming all the ULTs are accessing them by their index in the vector rather than by an ABT_mutex opaque pointer, then this should be fine as long as I am allowed to move ABT_mutex_memory around (regardless of their state). Is that the case? Same question for condition variables? Thanks, Matthieu

2 5

Idle Execution Stream
by Houjun Tang 10 May '21

10 May '21

Hi, >From what I understand, an Argobots execution stream is mapped to a pthread, and the ULTs are executed by it. What happens when there is no ULT in the ES, does the pthread sleep and wait on a conditional variable until a new ULT is scheduled for execution? Does it compete with other non-Argobots threads for the CPU resources while idle? -- Thanks, Houjun Tang

3 2

modifying scheduler event frequency?
by Phil Carns 21 Apr '21

21 Apr '21

Hi all, Is there a clean way to change a scheduler's event frequency on the fly? Browsing the API, I see two possibilities: * set it when the scheduler is first created (using ABT_sched_basic_freq?) * set it dynamically by manipulating the ABT_sched_get_data() pointer, but this seems especially dangerous since the sched data struct definition isn't public (i.e. it could cause memory corruption if the internal struct def changed) For some context (in case there is a different way to go about this entirely), I'm trying to figure out how to get ABT_info_trigger_print_all_thread_stacks() to print information more quickly, which IIUC relies on getting the active schedulers to call get_events() sooner. I'm happy to add some explicit ABT_thread_yield() shortly after the ABT_info_trigger_print_all_thread_stacks() to at least get the calling ES to execute it's scheduler loop immediately, but I think that won't matter much if it doesn't trip the frequency counter when I do it. Without this (at least with the _wait scheduler and threads that are occasionally tied up in system calls) I think the stack dump is likely to trigger too late to display what I'm hoping to capture when I call it. The first example I tried appeared to essentially defer dump until shutdown. thanks! -Phil

2 7

dumping stack information for running ULTs
by Phil Carns 16 Apr '21

16 Apr '21

Hi all, I followed Shintaro's suggestions from the previous list email thread and was able to get a stack dump routine that works as expected. For all pools under my control I can see stack information like this for each ULT, with stack unwinding: == pool (0x557d5b8c4580) == === ULT (0x7f47429590c1) === id : 0 ctx : 0x7f4742959120 p_ctx : 0x7f4742958fe0 p_link : (nil) stack : 0x7f47427590c0 stacksize : 2097152 #0 0x7f4744461760 in ythread_unwind_stack () <+16> (RSP = 0x7f4742959010) #1 0x7f474445e3dd in ABT_thread_yield () <+157> (RSP = 0x7f4742959020) #2 0x7f474447e255 in __margo_hg_progress_fn () <+645> (RSP = 0x7f4742959040) #3 0x7f474446381e in ABTD_ythread_func_wrapper () <+30> (RSP = 0x7f47429590b0) #4 0x7f47444639c1 in make_fcontext () <+33> (RSP = 0x7f47429590c0) 00007f47427590c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 00007f47427590e0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ... That only works for ULTs that are literally in the pool, though, I think. I don't believe that I am getting information for ULTs that are executing (and thus are not presently in a pool data structure). Is there any way to accomplish that? I tried to at least get the caller's own stack at least by doing something like this: ABT_thread_self(&self); ABT_info_print_thread_stack(outfile, self); That almost works, but I'm just getting the raw address information and not the translated stack unwinding: p_ctx : 0x7f4742758fb8 p_link : (nil) stack : 0x7f4742559000 stacksize : 2097152 00007f4742559000: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 00007f4742559020: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ... For completeness it would be nice if the caller's stack were also human readable, and even better if I could somehow find stack information for ULTs executing on other ESs as well. For our use case we typically spawn many detached threads into a service pool, so we aren't tracking thread references. thanks, -Phil

2 2

Announcing the release of Argobots 1.1
by Iwasaki, Shintaro 01 Apr '21

01 Apr '21

Dear Argobots Users and Developers, We are excited to announce that a new stable release of Argobots 1.1 is now available for download. Argobots 1.1 added numerous features such as tool interface, extended work-unit data access routines, improved affinity interface, and stack unwinding for debugging. Argobots 1.1 fixed several bugs and clarified the behavior of each API. Production environments are encouraged to upgrade to this release. All the changes from Argobots 1.0 are summarized here: https://www.argobots.org/changes/argobots_1_1/ Please find the tarball on our downloads page: https://www.argobots.org/downloads/ We thank all the Argobots users and developers for supporting us. Best Regards, The Argobots team =============================================================================== Changes in 1.1 =============================================================================== - Added a tool interface to support a profiler (see ABTX_prof) - Supported the stack unwinding feature - Implemented static initializers for ABT_mutex and ABT_cond - Improved work-unit-local storage support - Added several utility functions - Improved support for affinity settings via an environmental variable - Enhanced memory management of ULT stacks and descriptors - Used futex when external threads call Argobots synchronization operations - Revised the API specification - Added support for XLC, PGI, Solaris, and ARM HPC compilers - Supported Autoconf 2.70 - Fixed several bugs

1 0

Announcing the release of Argobots 1.1rc2
by Iwasaki, Shintaro 26 Mar '21

26 Mar '21

Dear Argobots Users and Developers, A new preview release of 1.1rc2 is now available for download. Argobots 1.1rc2 fixed major bugs in Argobots 1.1rc1 related to user-defined pools and ABT_xstream_set_main_sched(). This release also adds new utility functions (see https://github.com/pmodels/argobots/pull/319) Please find the tarball on our downloads page (https://www.argobots.org/downloads/) We plan to release Argobots 1.1 as scheduled (i.e., March 31st) unless we find another major issue. If you find any issues, please let us know by creating a GitHub issue. We thank all the Argobots users and developers for supporting us. Best Regards, The Argobots team

1 0

Re: [argobots-discuss] Program aborting in ABT_finalize (ABTI_pool_release failed)
by Dorier, Matthieu 23 Mar '21

23 Mar '21

Hi Shintaro, I think I’m narrowing down the problem (I’m testing right now). We initially had all our pools created with automatic=TRUE in Margo, and weren’t calling ABT_pool_free anywhere. But a recent commit from Phil enabled the use of a custom pool implementation, created with ABT_pool_create. The problem with these pools is that there is no “automatic” flag, so he changed to automatic=FALSE for the basic pools so he could call ABT_pool_free on all the pools, however not keeping track nor checking that a pool was associated with the primary ES. I think it would be good if Argobots allowed pools created by ABT_pool_create to have an “automatic” flag as well, and be free automatically. Otherwise I’m not even sure how we are supposed to free the pool associated with the primary ES if that pool happens to be a custom one. Thanks, Matthieu From: "Iwasaki, Shintaro" <siwasaki(a)anl.gov> Date: Tuesday, 23 March 2021 at 14:44 To: "Dorier, Matthieu" <mdorier(a)anl.gov> Subject: Re: Program aborting in ABT_finalize (ABTI_pool_release failed) Hi Matthieu, Even if automatic is false, please do not call ABT_pool_free() for the main ES's pool. It must be automatically freed by ABT_finalize(). Argobots 1.0 "works" though a few hundred bytes will be leaked because of its bug (a few hundred bytes per pool on ABT_finalize() as far as I checked with Valgrind). Argobots 1.1 (or the Argobots main branch) has fixed this memory leak issue, so I would recommend you use the latest one if possible. > Argobots isn’t checking whether a pool is in use before freeing it whether it’s used by the first ES. Currently Argobots checks neither of them, but it should be added to ease debugging. I created an issue: https://github.com/pmodels/argobots/issues/315 Thanks, Shintaro ________________________________ From: Dorier, Matthieu <mdorier(a)anl.gov> Sent: Tuesday, March 23, 2021 9:20 AM To: Iwasaki, Shintaro <siwasaki(a)anl.gov>; discuss(a)argobots.org <discuss(a)argobots.org>; discuss(a)lists.argobots.org <discuss(a)lists.argobots.org> Subject: Re: Program aborting in ABT_finalize (ABTI_pool_release failed) I’m creating the pool using ABT_pool_create_basic and “automatic” set to false, and I do indeed call ABT_pool_free on it. I guess Argobots isn’t checking whether a pool is in use before freeing it (or whether it’s used by the first ES)? My code is pretty complex so I tried to write a reproducer but I don’t see the error with it. Maybe I’m corrupting memory, somehow, and it just happens to be fine in small code that don’t reuse the memory after the ABT_pool_free… (though I ran address sanitizer and didn’t see any invalid access). I’ll see what happens if I remove the ABT_pool_free. Thanks, Matthieu From: "Iwasaki, Shintaro" <siwasaki(a)anl.gov> Date: Tuesday, 23 March 2021 at 14:01 To: "discuss(a)argobots.org" <discuss(a)argobots.org>, "discuss(a)lists.argobots.org" <discuss(a)lists.argobots.org> Cc: "Dorier, Matthieu" <mdorier(a)anl.gov> Subject: Re: Program aborting in ABT_finalize (ABTI_pool_release failed) Hi Matthieu, I don't know when such an error typically happens. I'd be happy if you could share your code with me so that I can closely look at it. This is just my guess, but perhaps ABT_pool_free() is used; in Argobots, primary ES's pool and schedulers may not be freed before ABT_finalize(). If you free that fifo_wait MPMC pool before ABT_finalize(), an already freed pool is accessed in ABT_finalize(), which may cause the error you wrote. (It is related to https://github.com/pmodels/argobots/issues/58) I attached a sample code to explain this issue. Please read the comments in the code if you are interested in it. Thanks, Shintaro ________________________________ From: Dorier, Matthieu via discuss <discuss(a)lists.argobots.org> Sent: Tuesday, March 23, 2021 6:56 AM To: discuss(a)argobots.org <discuss(a)argobots.org> Cc: Dorier, Matthieu <mdorier(a)anl.gov> Subject: [argobots-discuss] Program aborting in ABT_finalize (ABTI_pool_release failed) Hi, I’m getting this error in a program, when calling ABT_finalize: ../src/include/abti_pool.h:238: ABTI_pool_release: Assertion `ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0' failed. The program only uses the primary ES, it does not create additional ES, however it does replace the primary ES’s pool and scheduler by creating a fifo_wait MPMC pool and passing it to ABT_xstream_set_main_sched_basic, with ABT_SCHED_BASIC_WAIT as sched_predef. I’m using Argobots 1.0. I haven’t tested with other versions of Argobots. I’m sure I must be doing something wrong somewhere in my code. Could you tell me what I could look for that would lead to such an error? For additional information, here it the stack trace when the error happens: #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007f56797fe535 in __GI_abort () at abort.c:79 #2 0x00007f56797fe40f in __assert_fail_base (fmt=0x7f5679960ee0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7f5679d0e4c8 "ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0", file=0x7f5679d0e44b "../src/include/abti_pool.h", line=238, function=<optimized out>) at assert.c:92 #3 0x00007f567980c102 in __GI___assert_fail (assertion=0x7f5679d0e4c8 "ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0", file=0x7f5679d0e44b "../src/include/abti_pool.h", line=238, function=0x7f5679d0e660 <__PRETTY_FUNCTION__.6315> "ABTI_pool_release") at assert.c:101 #4 0x00007f5679d09e5e in ABTI_sched_free.part.4 () from /projects/spack/opt/spack/linux-debian10-sandybridge/gcc-8.3.0/argobots-1.0-e4x7h6mgt7igmocc4ajro2g743ej322m/lib/libabt.so.0 #5 0x00007f5679cfe2c5 in ABTI_xstream_free () from /projects/spack/opt/spack/linux-debian10-sandybridge/gcc-8.3.0/argobots-1.0-e4x7h6mgt7igmocc4ajro2g743ej322m/lib/libabt.so.0 #6 0x00007f5679cfa080 in ABT_finalize () Thanks, Matthieu

2 2

Program aborting in ABT_finalize (ABTI_pool_release failed)
by Dorier, Matthieu 23 Mar '21

23 Mar '21

Hi, I’m getting this error in a program, when calling ABT_finalize: ../src/include/abti_pool.h:238: ABTI_pool_release: Assertion `ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0' failed. The program only uses the primary ES, it does not create additional ES, however it does replace the primary ES’s pool and scheduler by creating a fifo_wait MPMC pool and passing it to ABT_xstream_set_main_sched_basic, with ABT_SCHED_BASIC_WAIT as sched_predef. I’m using Argobots 1.0. I haven’t tested with other versions of Argobots. I’m sure I must be doing something wrong somewhere in my code. Could you tell me what I could look for that would lead to such an error? For additional information, here it the stack trace when the error happens: #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007f56797fe535 in __GI_abort () at abort.c:79 #2 0x00007f56797fe40f in __assert_fail_base (fmt=0x7f5679960ee0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7f5679d0e4c8 "ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0", file=0x7f5679d0e44b "../src/include/abti_pool.h", line=238, function=<optimized out>) at assert.c:92 #3 0x00007f567980c102 in __GI___assert_fail (assertion=0x7f5679d0e4c8 "ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0", file=0x7f5679d0e44b "../src/include/abti_pool.h", line=238, function=0x7f5679d0e660 <__PRETTY_FUNCTION__.6315> "ABTI_pool_release") at assert.c:101 #4 0x00007f5679d09e5e in ABTI_sched_free.part.4 () from /projects/spack/opt/spack/linux-debian10-sandybridge/gcc-8.3.0/argobots-1.0-e4x7h6mgt7igmocc4ajro2g743ej322m/lib/libabt.so.0 #5 0x00007f5679cfe2c5 in ABTI_xstream_free () from /projects/spack/opt/spack/linux-debian10-sandybridge/gcc-8.3.0/argobots-1.0-e4x7h6mgt7igmocc4ajro2g743ej322m/lib/libabt.so.0 #6 0x00007f5679cfa080 in ABT_finalize () Thanks, Matthieu

2 2