Is the issue that when a ULT is created it draws from the stack queue on the calling ES, but when the ULT is free'd it is put on the stack queue of the ES that executed it? In the example I have that is consuming more memory than expected, the ULTs are always dispatched by one specific ES (the one that is driving progress on the network transport) and executed on a different ES (the dedicated pool set aside for servicing I/O operations). thanks, -Phil On 02/23/2018 01:09 PM, Phil Carns wrote:
Hi all,
I'm trying to debug problem with a service that spawns a new ULT for each incoming client request. The service works correctly, but memory consumption is considerably higher over time if I create a dedicated pool with it's own execution streams to run the ULTs instead of just running them on the main ES.
There are some details here: https://xgitlab.cels.anl.gov/sds/margo/issues/40
That specific test example is creating a total of 40,000 ULTs over the course of execution. It doesn't have that many active concurrently, though. The client program is issuing no more than 4 concurrent requests. A ULT will slightly outlive the request handler, but the number of active ULTS isn't growing indefinitely. I can confirm that argobots is freeing them throughout execution if I turn on Argobots logging.
I can see in the Argobots source code that when a ULT is freed it doesn't literally free() the stack memory for that ULT; it is kept on a queue:
https://github.com/pmodels/argobots/blob/master/src/include/abti_mem.h#L287
If I print the value of num_stacks in Argobots at the above line, I get the following:
- when using the default/main ES: no higher than 512
- when explicitly creating a new pools and ES: 40,000
I haven't tracked down the logic to understand why there is a difference here, but is this expected? It looks like Argobots is keeping a much larger number of stacks around (without reusing them?) if I create a new pool and ES to run my ULTs.
thanks,
-Phil
_______________________________________________ discuss mailing list [email protected] https://lists.argobots.org/mailman/listinfo/discuss