Hi, I get a "No free event buffers.Try increasing via g_tw_events_per_pe or --extramem" error in tw_event_new when running some not too large scale simulation, say 12840 LPs. I add option "--extramem=8192" and it doesn't help. And there is still 90GB of memory left. So, any general suggestion of avoiding this error? Thanks.
Hi Jian, The ROSS simulations start up with a certain number of events statically allocated per PE (The message-size parameter in the CODES config files tell ROSS how big an event is in bytes). At some point during the simulation, if the events being processed by a PE exceed the number of events that were allocated at the simulation start up, this error is thrown and the simulation terminates. To handle this issue, you could do a —extramem that you already tried. This will allocate an extra number of events per PE. You could try increasing extramem till the point where the event memory is sufficient. However, when this error pops up, it is usually better to double check the model behavior I.e. If the model is issuing a reasonable number of events, it isn't running into an infinite loop of event generation or if the workload is reasonable (it could be that the workload is injecting a very large number of events in a short amount of time). Also, if you are running in optimistic mode and you are getting too many rollbacks, that could be another reason of the event memory running out. So it will help to run the simulation in sequential mode first and see if the error is still there. Regards, Misbah From: <[email protected]<mailto:[email protected]>> on behalf of Jian Peng <[email protected]<mailto:[email protected]>> Date: Wednesday, April 5, 2017 at 12:48 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [codes-ross-users] "No free event buffers" error Hi, I get a "No free event buffers.Try increasing via g_tw_events_per_pe or --extramem" error in tw_event_new when running some not too large scale simulation, say 12840 LPs. I add option "--extramem=8192" and it doesn't help. And there is still 90GB of memory left. So, any general suggestion of avoiding this error? Thanks.
Thanks for the explanation! I'll try it out. On Wed, Apr 5, 2017 at 11:18 AM, Mubarak, Misbah <[email protected]> wrote:
Hi Jian,
The ROSS simulations start up with a certain number of events statically allocated per PE (The message-size parameter in the CODES config files tell ROSS how big an event is in bytes). At some point during the simulation, if the events being processed by a PE exceed the number of events that were allocated at the simulation start up, this error is thrown and the simulation terminates.
To handle this issue, you could do a —extramem that you already tried. This will allocate an extra number of events per PE. You could try increasing extramem till the point where the event memory is sufficient. However, when this error pops up, it is usually better to double check the model behavior I.e. If the model is issuing a reasonable number of events, it isn't running into an infinite loop of event generation or if the workload is reasonable (it could be that the workload is injecting a very large number of events in a short amount of time).
Also, if you are running in optimistic mode and you are getting too many rollbacks, that could be another reason of the event memory running out. So it will help to run the simulation in sequential mode first and see if the error is still there.
Regards, Misbah
From: <[email protected]> on behalf of Jian Peng <[email protected]> Date: Wednesday, April 5, 2017 at 12:48 PM To: "[email protected]" <[email protected]. anl.gov> Subject: [codes-ross-users] "No free event buffers" error
Hi,
I get a "No free event buffers.Try increasing via g_tw_events_per_pe or --extramem" error in tw_event_new when running some not too large scale simulation, say 12840 LPs. I add option "--extramem=8192" and it doesn't help. And there is still 90GB of memory left.
So, any general suggestion of avoiding this error? Thanks.
Another way to control ROSS’s excessive memory consumption is the max-opt-lookahead option: https://github.com/carothersc/ROSS/commit/53b7fcabcb9b8c015b1f17e3ce1a94298b... <https://github.com/carothersc/ROSS/commit/53b7fcabcb9b8c015b1f17e3ce1a94298b06920d> I use it often and this helps avoid running out of memory. --- Nikhil Jain Postdoctoral Fellow, Lawrence Livermore National Laboratory [email protected], http://nikhil-jain.github.io/
On Apr 5, 2017, at 13:34, Jian Peng <[email protected]> wrote:
Thanks for the explanation! I'll try it out.
On Wed, Apr 5, 2017 at 11:18 AM, Mubarak, Misbah <[email protected] <mailto:[email protected]>> wrote: Hi Jian,
The ROSS simulations start up with a certain number of events statically allocated per PE (The message-size parameter in the CODES config files tell ROSS how big an event is in bytes). At some point during the simulation, if the events being processed by a PE exceed the number of events that were allocated at the simulation start up, this error is thrown and the simulation terminates.
To handle this issue, you could do a —extramem that you already tried. This will allocate an extra number of events per PE. You could try increasing extramem till the point where the event memory is sufficient. However, when this error pops up, it is usually better to double check the model behavior I.e. If the model is issuing a reasonable number of events, it isn't running into an infinite loop of event generation or if the workload is reasonable (it could be that the workload is injecting a very large number of events in a short amount of time).
Also, if you are running in optimistic mode and you are getting too many rollbacks, that could be another reason of the event memory running out. So it will help to run the simulation in sequential mode first and see if the error is still there.
Regards, Misbah
From: <[email protected] <mailto:[email protected]>> on behalf of Jian Peng <[email protected] <mailto:[email protected]>> Date: Wednesday, April 5, 2017 at 12:48 PM To: "[email protected] <mailto:[email protected]>" <[email protected] <mailto:[email protected]>> Subject: [codes-ross-users] "No free event buffers" error
Hi,
I get a "No free event buffers.Try increasing via g_tw_events_per_pe or --extramem" error in tw_event_new when running some not too large scale simulation, say 12840 LPs. I add option "--extramem=8192" and it doesn't help. And there is still 90GB of memory left.
So, any general suggestion of avoiding this error? Thanks.
_______________________________________________ codes-ross-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/codes-ross-users
Great, thanks! On Wed, Apr 5, 2017 at 2:48 PM, Nikhil Jain <[email protected]> wrote:
Another way to control ROSS’s excessive memory consumption is the max-opt-lookahead option:
https://github.com/carothersc/ROSS/commit/53b7fcabcb9b8c015b1f17e3ce1a94 298b06920d
I use it often and this helps avoid running out of memory.
--- Nikhil Jain Postdoctoral Fellow, Lawrence Livermore National Laboratory [email protected], http://nikhil-jain.github.io/
On Apr 5, 2017, at 13:34, Jian Peng <[email protected]> wrote:
Thanks for the explanation! I'll try it out.
On Wed, Apr 5, 2017 at 11:18 AM, Mubarak, Misbah <[email protected]> wrote:
Hi Jian,
The ROSS simulations start up with a certain number of events statically allocated per PE (The message-size parameter in the CODES config files tell ROSS how big an event is in bytes). At some point during the simulation, if the events being processed by a PE exceed the number of events that were allocated at the simulation start up, this error is thrown and the simulation terminates.
To handle this issue, you could do a —extramem that you already tried. This will allocate an extra number of events per PE. You could try increasing extramem till the point where the event memory is sufficient. However, when this error pops up, it is usually better to double check the model behavior I.e. If the model is issuing a reasonable number of events, it isn't running into an infinite loop of event generation or if the workload is reasonable (it could be that the workload is injecting a very large number of events in a short amount of time).
Also, if you are running in optimistic mode and you are getting too many rollbacks, that could be another reason of the event memory running out. So it will help to run the simulation in sequential mode first and see if the error is still there.
Regards, Misbah
From: <[email protected]> on behalf of Jian Peng <[email protected]> Date: Wednesday, April 5, 2017 at 12:48 PM To: "[email protected]" <[email protected] l.gov> Subject: [codes-ross-users] "No free event buffers" error
Hi,
I get a "No free event buffers.Try increasing via g_tw_events_per_pe or --extramem" error in tw_event_new when running some not too large scale simulation, say 12840 LPs. I add option "--extramem=8192" and it doesn't help. And there is still 90GB of memory left.
So, any general suggestion of avoiding this error? Thanks.
_______________________________________________ codes-ross-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/codes-ross-users
participants (3)
-
Jian Peng -
Mubarak, Misbah -
Nikhil Jain