- discuss - mailman3.cels.anl.gov

Removed Functions in Argobots 1.0
by Shintaro Iwasaki 23 Mar '20

23 Mar '20

Dear Argobots Users and Developers, Hello. Finally, we will release Argobots 1.0 in a few weeks. Argobots 1.0 will not support the following deprecated functions and constants: retain/release: (https://github.com/pmodels/argobots/pull/138) ABT_thread_retain(), ABT_thread_release(), ABT_task_retain(), and ABT_task_release(). Reason: fundamentally, there is no good way to fix bugs in these functions without sacrificing performance. xstream: (https://github.com/pmodels/argobots/pull/147) ABT_xstream_start(), ABT_XSTREAM_STATE_CREATED, and ABT_XSTREAM_STATE_READY Reason: CREATED and READY states are merged to RUNNING. ABT_xstream_start(), which takes a READY execution stream, becomes useless. event: (https://github.com/pmodels/argobots/pull/144) ABT_event_add_callback(), ABT_event_del_callback(), ABT_event_prof_start(), ABT_event_prof_stop(), ABT_event_prof_publish(), ... Reason: event features (energy-efficient scheduling and scheduler profiling) should not be integrated into Argobots; they should be implemented on top of it. Please check whether your programs are using them. If these functions are really necessary, please let us know so that we can offer alternative solutions or revert these changes. Please report it by writing a comment at the following (https://github.com/pmodels/argobots/issues/156) or reply to this mail. I am deeply sorry for already merging these changes to the master branch. We will inform you of API changes ahead of time. If you have any questions, please feel free to ask me. Best Regards, Shintaro Iwasaki

1 0

Re: [argobots-discuss] Argobots ABT_eventual_set too slow
by Iwasaki, Shintaro 25 Apr '19

25 Apr '19

Hi, Houjun, Please use ABT_xstream_set_cpubind or ABT_xstream_set_affinity. You can also disable the xstream affinity by setting ABT_SET_AFFINITY=no. Thanks, Shintaro On Thu, Apr 25, 2019, 18:08 Shintaro Iwasaki <wleaf.iwashi(a)gmail.com<mailto:[email protected]>> wrote: Hi, Houjun, Please use ABT_xstream_set_cpubind or ABT_xstream_set_affinity. You can also disable the xstream affinity by setting ABT_SET_AFFINITY=no. Thanks, Shintaro On Thu, Apr 25, 2019 at 5:27 PM Houjun Tang via discuss <discuss(a)lists.argobots.org<mailto:[email protected]>> wrote: Hi Pavan, Yes, the argobots and execution are under my control, but I'm having issues to use pthread_setaffinity_np(), I added some code from the example in http://man7.org/linux/man-pages/man3/pthread_getaffinity_np.3.html, but it seems it is only supported by the GNU compilers as I have to add "#define _GNU_SOURCE" to the beginning of the code. But the application is usually compile with Intel compilers on NERSC machines, so it won't work in that case. I'm not very familiar with the thread binding related things, is there any other ways to do it? Thanks, Houjun Tang On Thu, Apr 25, 2019 at 1:24 PM Balaji, Pavan <balaji(a)anl.gov<mailto:[email protected]>> wrote: Hi Houjun, > On Apr 25, 2019, at 3:19 PM, Houjun Tang <htang4(a)lbl.gov<mailto:[email protected]>> wrote: > I have tried different things as you suggested, and it seems setting OMP_PROC_BIND=false (previously it is set to "spread") can solve the problem. With that variable set, ABT_eventual_set() takes similar time (under 0.00001 seconds) to when OpenMP is disabled. So maybe it's because the Argobots threads and OpenMP threads are executing on the same core and causes the slowdown? That's a good find. Yes, it looks like the OpenMP runtime is forcing the pthread to a specific core, so any child threads created by that thread would also be bound to the same core. Presumably the argobots thread and execution stream are under your control. Can you unset the binding in your execution stream when you create it? You can use pthread_setaffinity_np, although that needs to be executed inside your ULT. -- Pavan _______________________________________________ discuss mailing list discuss(a)lists.argobots.org<mailto:[email protected]> https://lists.argobots.org/mailman/listinfo/discuss

1 0

Argobots ABT_eventual_set too slow
by Houjun Tang 25 Apr '19

25 Apr '19

Hi, I'm using Argobots as the engine for executing asynchronous I/O operations in the background of an HDF5 application, but found it to be slow in some operations. With profiling, the slowdown comes mostly from *ABT_eventual_set*. Below is a boxplot of the ABT_eventual_set time (measured by calling gettimeofday before and after it) from 385 operations, running with one process and one Argobots thread. The *_fn are different functions executed by Argobots. In most cases it's below 0.1s, but there are several cases that are taking more than 0.25 seconds. As these HDF5 operations take less than 0.1 seconds, the overhead of ABT_eventual_set becomes dominant. Any idea what could have caused this? Thanks, Houjun Tang [image: image.png]

5 20

Take a task out of an Argobot pool?
by Quincey Koziol 28 Feb '19

28 Feb '19

Hi all, I need to pull a task out of an Argobot pool and execute it “now”, in the current thread. Any suggestions for the best way to do that? I am thinking about calling ABT_task_cancel() to get it out of the pool, then just call the task_func (that we gave to ABT_task_create) directly from the place we took it out of the pool. Will that work? Is there a better way? Thanks, Quincey

2 2

how to debug a stack overrun in Argobots
by Carns, Philip H. 27 Feb '19

27 Feb '19

Hi all, There is a little bit of back story on https://github.com/pmodels/argobots/issues/93 , but make a long story short we have realized that we have some code that is overflowing the stack in Argobots. Many thanks to Shintaro for his help and insight or we may have never figured this out. We can work around the problem with `export ABT_THREAD_STACKSIZE=$((1024 * 1024))`. This not only fixes a Power8 test case for us, but also appears to solve a different frustrating, nonsensical segmentation fault that we've been chasing with a different code permutation on x86_64. Any suggestions on how to track down what's triggering this in our code or get a better idea of how much stack we need? We are using a considerable number of libraries, many of which are not maintained by us, so I don't even know where to start looking yet. My usual go to tool for this would be asan in gcc or clang, but I don't think that will work correctly with Argobots, and maybe there is a better solution anyway. thanks, -Phil

3 7

ULT scheduler switching to Tasklet
by POLYKARPOS THOMADAKIS 07 Dec '18

07 Dec '18

Hello, I have observed the following behavior when I have a custom scheduler that runs a Tasklet: I retrieve the type of the scheduler before I run the Tasklet unit popped from the argobots pool and it is ULT which is what expected. The scheduler runs the Tasklet, and after it's done I retrieve its type again and it has changed to Tasklet which is unexpected. This seems a bit weird for me and I would like to know whether this shall be the functionality. I also checked against an older version of argobots and I found out that this is not happening there. Specifically, in version 1.0a1 the function ABTI_xstream_schedule_task <https://press3.mcs.anl.gov//wp-content/blogs.dir/231/files/argobots/doxygen…> saves the state of current thread/task and recovers it after it's done executing the Tasklet. This is removed in current version. Thank you for your time, Polykarpos

2 1

Announcing the v1.0rc1 releases of BOLT and Argobots
by Amer, Abdelhalim 09 Nov '18

09 Nov '18

All, The BOLT/Argobots team is pleased to announce the availability of new release versions of BOLT and Argobots (bolt-v1.0rc1 and argobots-v1.0rc1). These contain several new features, and performance optimizations, and bug fixes over the previous 1.0b1 releases. Among the biggest features are the BOLT upgrade to LLVM OpenMP 7.0, supporting most of OpenMP 4.5, and production integration of the new Argobots dynamic promotion technique (which will presented at SC'18). We encourage users to upgrade to these latest releases. Tarballs can be found either on the BOLT and Argobots websites (bolt-omp.org and argobots.org) on GitHub (pmodels/bolt and pmodels/argobots), or on the SOLLVE website. Best regards, The BOLT/Argobots Team

1 0

Possible issue
by POLYKARPOS THOMADAKIS 01 Nov '18

01 Nov '18

Hello there, I am experiencing the following issue using Argobots, I'm not sure if that's a bug or an assumption made by Argobots that is not explicit. Here is the issue: I have a set of custom pools, 1 per Execution Stream. In each of the pools, only the associated execution stream can push while other streams can grab a unit from other streams' pool safely. Those are the characteristics of an SPMC pool, so that's the type I specify on their creation. The underlying data structure is a lock-free cyclic deque where the pushing is done at the bottom, the popping (where the owner stream of the pool grabs a unit) also at the bottom, and the stealing (where a stream grabs a unit from the pools of another stream) from the top. In this way when a ULT is created it's always pushed on the pool of the creating stream and can be stolen by other streams safely. The application works perfectly with just creating and executing ULTs, however, when I need the join functionality is where the problem occurs. The workflow is as follows: 1) ULT 0 spawns its children ULTs in ES 0 2) ES 1 steals one (or more) of the ULT from ES 0 3) ULT 0 on ES 0 joins one of its children -> Argobots suspends its execution 4) ES 1 terminates with the execution of the stolen ULT. This will cause Argobots to try to awaken the blocked ULT 0 by pushing it back to the pool of its last stream, which is ES 0. And here is the problem, ES 1 awakens ULT 0, pushes it to the pool of ES 0, thus, breaking the rule that only the associated stream of a pool is allowed to push to it. Since the user has no API to defined to which pool a unit shall be pushed in such situations, I believed that by setting the pool type to SPMC Argobots would take care of this. The lines that produce this issue: arch/abtd_thread.c:88 -> Terminating thread awakens blocked joiner thread.c:2017 -> Joiner is pushed to the last pool it ran before blocking, causing one stream to push to the pools of another My question is whether this behavior is the expected one or not. In other words, if the user is expected to take into consideration this behavior when designing his/her custom pools. Sorry for the long email and thank you for your time. Best, Polykarpos

2 2

Dumping ULT stacks
by Iwasaki, Shintaro 07 Sep '18

07 Sep '18

Hi, Johann, Thank you for suggestion. Currently, Argobots does not have functions printing ULT stacks. We don't know someone implemented things similar. However, stack dump should be very useful for debugging. Since it is needed, we will raise the priority of this issue. In github, the similar issue has been posted. https://github.com/pmodels/argobots/issues/12 I will add a technical comment about this issue soon. For example, printing all ULTs in a pool can be technically difficult because there is no general way to get all the ULTs in the pool. We would be happy if you would join the discussion. Best, Shintaro Iwasaki From: "Lombardi, Johann" <johann.lombardi(a)intel.com> Subject: [argobots-discuss] Dumping ULT stacks Date: September 7, 2018 at 9:50:25 AM CDT To: "discuss(a)lists.argobots.org" <discuss(a)lists.argobots.org> Cc: "Faccini, Bruno" <bruno.faccini(a)intel.com> Reply-To: <discuss(a)lists.argobots.org> Hi there, For debugging purpose, we would like to dump all ULT stacks when our service (using multiple xstreams and mulitple pools) receives SIGUSR1. We found ABT_info_print_all_xstreams(), but it does not print any ULT stacks. Has anyone already implemented something similar? Thanks in advance. Cheers, Johann --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ discuss mailing list discuss(a)lists.argobots.org https://lists.argobots.org/mailman/listinfo/discuss

1 0

Dumping ULT stacks
by Lombardi, Johann 07 Sep '18

07 Sep '18

Hi there, For debugging purpose, we would like to dump all ULT stacks when our service (using multiple xstreams and mulitple pools) receives SIGUSR1. We found ABT_info_print_all_xstreams(), but it does not print any ULT stacks. Has anyone already implemented something similar? Thanks in advance. Cheers, Johann --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0