FWIW, we use the ABT_POOL_FIFO_WAIT in the Mochi project as our default scheduler.  It works well as a tradeoff between CPU usage and response time. That pool must also be used in conjunction with the ABT_SCHED_BASIC_WAIT scheduler to get the full benefit.  There is a simple example of setting both the waiting scheduler and pool at https://github.com/mochi-hpc/mochi-abt-io/blob/main/src/abt-io.c#L1351. That example is for a dedicated (explicitly created) pool, but you can also do something similar for the implicit main execution stream as well if necessary using ABT_xstream_set_main_sched_basic(). -Phil On 5/7/21 6:45 PM, Iwasaki, Shintaro via discuss wrote:

Hello Houjun,

Thanks for your question. It depends on the setting. By default, the execution stream will busily check the pool, so it wastes its underlying core if the pool is empty. If so, it competes with other non-Argobots threads.

To let idle execution streams sleep, Argobots provides a few options:

1. If you enable the scheduler sleep at configuration time, the ES that uses a builtin scheduler will sleep when it cannot consecutively find a ready ULT in its associated pools. ./configure --enable-sched-sleep The sleep time is configurable via ABT_SCHED_SLEEP_NSEC=XXX ABT_SCHED_SLEEP_NSEC=100 ./a.out This implementation is very naive but it is easy for users to try.

2. If ABT_POOL_FIFO_WAIT is used to create a pool, the ES will timed-wait on an empty pool until a ULT is pushed to that pool. Internally, it uses a condition variable, so it is smarter. This needs some efforts to use (at least you need to change the code). ... Perhaps you might be interested in the following example. https://github.com/pmodels/argobots/blob/main/examples/profiling/async_engin... <https://github.com/pmodels/argobots/blob/main/examples/profiling/async_engine.c> The following PR is related. This (indirectly) shows how to check the CPU resource consumption by ESs. https://github.com/pmodels/argobots/pull/208 <https://github.com/pmodels/argobots/pull/208>

If you have any further questions, please feel free to ask!

Thanks, Shintaro

------------------------------------------------------------------------ *From:* Houjun Tang via discuss <[email protected]> *Sent:* Friday, May 7, 2021 5:25 PM *To:* [email protected] <[email protected]> *Cc:* Houjun Tang <[email protected]> *Subject:* [argobots-discuss] Idle Execution Stream Hi,

From what I understand, an Argobots execution stream is mapped to a pthread, and the ULTs are executed by it. What happens when there is no ULT in the ES, does the pthread sleep and wait on a conditional variable until a new ULT is scheduled for execution? Does it compete with other non-Argobots threads for the CPU resources while idle?

-- Thanks, Houjun Tang

_______________________________________________ discuss mailing list [email protected] https://lists.argobots.org/mailman/listinfo/discuss