Hello there,
I am experiencing the following issue using Argobots, I'm not sure if
that's a bug or an assumption made by Argobots that is not explicit.
Here is the issue:
I have a set of custom pools, 1 per Execution Stream.
In each of the pools, only the associated execution stream can push while
other streams can grab a unit from other streams' pool safely.
Those are the characteristics of an SPMC pool, so that's the type I specify
on their creation.
The underlying data structure is a lock-free cyclic deque where the pushing
is done at the bottom, the popping (where the owner stream of the pool
grabs a unit)
also at the bottom, and the stealing (where a stream grabs a unit from the
pools of another stream) from the top.
In this way when a ULT is created it's always pushed on the pool of the
creating stream and can be stolen by other streams safely. The application
works perfectly
with just creating and executing ULTs, however, when I need the join
functionality is where the problem occurs. The workflow is as follows:
1) ULT 0 spawns its children ULTs in ES 0
2) ES 1 steals one (or more) of the ULT from ES 0
3) ULT 0 on ES 0 joins one of its children -> Argobots suspends its
execution
4) ES 1 terminates with the execution of the stolen ULT. This will cause
Argobots to try to awaken the blocked ULT 0 by pushing it back to the pool
of its last stream,
which is ES 0.
And here is the problem, ES 1 awakens ULT 0, pushes it to the pool of ES 0,
thus, breaking the rule that only the associated stream of a pool is
allowed to push to it.
Since the user has no API to defined to which pool a unit shall be pushed
in such situations, I believed that by setting the pool type to SPMC
Argobots would take care
of this.
The lines that produce this issue:
arch/abtd_thread.c:88 -> Terminating thread awakens blocked joiner
thread.c:2017 -> Joiner is pushed to the last pool it ran before blocking,
causing one stream to push to the pools of another
My question is whether this behavior is the expected one or not. In other
words, if the user is expected to take into consideration this behavior
when designing his/her
custom pools.
Sorry for the long email and thank you for your time.
Best,
Polykarpos
