- codes-ross-users - mailman3.cels.anl.gov

Re: [codes-ross-users] Replay HPL's dumpi trace on CODES
by Mubarak, Misbah 08 Jun '17

08 Jun '17

Hi Maxime, Thanks for your message. There seems to be a data type that is either not supported by DUMPI or CODES. Are you familiar with what data types are being used by the HPL trace? I will find out if the support for them can be added in the code. Regards, Misbah From: <codes-ross-users-bounces(a)lists.mcs.anl.gov<mailto:[email protected]>> on behalf of Maxime Chevalier <maxime.chevalier(a)inria.fr<mailto:[email protected]>> Date: Monday, May 29, 2017 at 3:51 AM To: "codes-ross-users(a)lists.mcs.anl.gov<mailto:[email protected]>" <codes-ross-users(a)lists.mcs.anl.gov<mailto:[email protected]>> Subject: [codes-ross-users] Replay HPL's dumpi trace on CODES Hi, I'm trying to replay HPL's DUMPI trace generated on my computer with CODES. Unfortunately, I get a lot of "Undefined data type" errors (see the trace below). I have already replayed AMG traces (downloaded here<http://portal.nersc.gov/project/CAL/designforward.htm>) and replayed my own generated AMG traces. It has worked fine. So I'm wondering if I did something wrong, or if it's HPL fault. Best regards, Maxime Trace : ROSS Revision: 4c6a7d8eb9c784797d900edfc76725d62ec25941 tw_net_start: Found world size to be 1 ROSS Core Configuration: Total Nodes 1 Total Processors [Nodes (1) x PE_per_Node (1)] 1 Total KPs [Nodes (1) x KPs (16)] 16 Total LPs 5 Simulation End Time 300000000000.00 LP-to-PE Mapping model defined ROSS Event Memory Allocation: Model events 1281 Network events 50000 Total events 51280 Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type *** START SEQUENTIAL SIMULATION *** *** END SIMULATION *** LP 1 unmatched irecvs 1 unmatched sends 0 Total sends 0 receives 1 collectives 0 delays 7 wait alls 0 waits 0 send time 0.000000 wait 0.000000 : Running Time = 0.0000 seconds TW Library Statistics: Total Events Processed 8 Events Aborted (part of RBs) 0 Events Rolled Back 0 Event Ties Detected in PE Queues 0 Efficiency 100.00 % Total Remote (shared mem) Events Processed 0 Percent Remote Events 0.00 % Total Remote (network) Events Processed 0 Percent Remote Events 0.00 % Total Roll Backs 0 Primary Roll Backs 0 Secondary Roll Backs 0 Fossil Collect Attempts 0 Total GVT Computations 0 Net Events Processed 8 Event Rate (events/sec) 307692.3 Total Events Scheduled Past End Time 0 TW Memory Statistics: Events Allocated 51281 Memory Allocated 51168 Memory Wasted 720 TW Data Structure sizes in bytes (sizeof): PE struct 608 KP struct 144 LP struct 128 LP Model struct 760 LP RNGs 80 Total LP 968 Event struct 144 Event struct with Model 928 TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz): Priority Queue (enq/deq) 0.0000 AVL Tree (insert/delete) 0.0000 LZ4 (de)compression 0.0000 Buddy system 0.0000 Event Processing 0.0000 Event Cancel 0.0000 Event Abort 0.0000 GVT 0.0000 Fossil Collect 0.0000 Primary Rollbacks 0.0000 Network Read 0.0000 Statistics Computation 0.0000 Statistics Write 0.0000 Total Time (Note: Using Running Time above for Speedup) 0.0001 TW GVT Statistics: MPI AllReduce GVT Interval 16 GVT Real Time Interval (cycles) 0 GVT Real Time Interval (sec) 0.00000000 Batch Size 16 Forced GVT 0 Total GVT Computations 0 Total All Reduce Calls 0 Average Reduction / GVT -nan Total bytes sent 0 recvd 4 max runtime 0.000000 ns avg runtime 0.000000 max comm time 0.000000 avg comm time -66232.000000 max send time 0.000000 avg send time 0.000000 max recv time 0.000000 avg recv time 0.000000 max wait time 0.000000 avg wait time 0.000000 LP-IO: writing output to hpl-trace-25282-1495543803/ LP-IO: data files: hpl-trace-25282-1495543803/mpi-replay-stats hpl-trace-25282-1495543803/model-net-category-all

2 5

Too many rollback events
by Jian Peng 07 Jun '17

07 Jun '17

Hi, My simulation is having too many events rolled back, like 70%. It's still relatively small scale, 16 cores on 4 nodes, using optimistic scheduler. The result is attached. Any general idea of reducing number of rollback events? I used the max-opt-lookahead option and set it to 10000. Is it still a large number? Thanks!

2 2

Replay HPL's dumpi trace on CODES
by Maxime Chevalier 29 May '17

29 May '17

Hi, I'm trying to replay HPL's DUMPI trace generated on my computer with CODES. Unfortunately, I get a lot of "Undefined data type" errors (see the trace below). I have already replayed AMG traces (downloaded here ) and replayed my own generated AMG traces. It has worked fine. So I'm wondering if I did something wrong, or if it's HPL fault. Best regards, Maxime Trace : <blockquote> ROSS Revision: 4c6a7d8eb9c784797d900edfc76725d62ec25941 tw_net_start: Found world size to be 1 ROSS Core Configuration: Total Nodes 1 Total Processors [Nodes (1) x PE_per_Node (1)] 1 Total KPs [Nodes (1) x KPs (16)] 16 Total LPs 5 Simulation End Time 300000000000.00 LP-to-PE Mapping model defined ROSS Event Memory Allocation: Model events 1281 Network events 50000 Total events 51280 Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type Undefined data type *** START SEQUENTIAL SIMULATION *** *** END SIMULATION *** LP 1 unmatched irecvs 1 unmatched sends 0 Total sends 0 receives 1 collectives 0 delays 7 wait alls 0 waits 0 send time 0.000000 wait 0.000000 : Running Time = 0.0000 seconds TW Library Statistics: Total Events Processed 8 Events Aborted (part of RBs) 0 Events Rolled Back 0 Event Ties Detected in PE Queues 0 Efficiency 100.00 % Total Remote (shared mem) Events Processed 0 Percent Remote Events 0.00 % Total Remote (network) Events Processed 0 Percent Remote Events 0.00 % Total Roll Backs 0 Primary Roll Backs 0 Secondary Roll Backs 0 Fossil Collect Attempts 0 Total GVT Computations 0 Net Events Processed 8 Event Rate (events/sec) 307692.3 Total Events Scheduled Past End Time 0 TW Memory Statistics: Events Allocated 51281 Memory Allocated 51168 Memory Wasted 720 TW Data Structure sizes in bytes (sizeof): PE struct 608 KP struct 144 LP struct 128 LP Model struct 760 LP RNGs 80 Total LP 968 Event struct 144 Event struct with Model 928 TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz): Priority Queue (enq/deq) 0.0000 AVL Tree (insert/delete) 0.0000 LZ4 (de)compression 0.0000 Buddy system 0.0000 Event Processing 0.0000 Event Cancel 0.0000 Event Abort 0.0000 GVT 0.0000 Fossil Collect 0.0000 Primary Rollbacks 0.0000 Network Read 0.0000 Statistics Computation 0.0000 Statistics Write 0.0000 Total Time (Note: Using Running Time above for Speedup) 0.0001 TW GVT Statistics: MPI AllReduce GVT Interval 16 GVT Real Time Interval (cycles) 0 GVT Real Time Interval (sec) 0.00000000 Batch Size 16 Forced GVT 0 Total GVT Computations 0 Total All Reduce Calls 0 Average Reduction / GVT -nan Total bytes sent 0 recvd 4 max runtime 0.000000 ns avg runtime 0.000000 max comm time 0.000000 avg comm time -66232.000000 max send time 0.000000 avg send time 0.000000 max recv time 0.000000 avg recv time 0.000000 max wait time 0.000000 avg wait time 0.000000 LP-IO: writing output to hpl-trace-25282-1495543803/ LP-IO: data files: hpl-trace-25282-1495543803/mpi-replay-stats hpl-trace-25282-1495543803/model-net-category-all </blockquote>

1 0

"No free event buffers" error
by Jian Peng 05 Apr '17

05 Apr '17

Hi, I get a "No free event buffers.Try increasing via g_tw_events_per_pe or --extramem" error in tw_event_new when running some not too large scale simulation, say 12840 LPs. I add option "--extramem=8192" and it doesn't help. And there is still 90GB of memory left. So, any general suggestion of avoiding this error? Thanks.

3 4

Fat-Tree Wiki Update
by noah wolfe 31 Mar '17

31 Mar '17

Hi All, The issue regarding the generation of static routing tables for the Fat-Tree network model has been resolved and the updated process has been posted to the wiki. Thanks Peixin for the fixes. Regards, Noah

1 0

how to use fattree static routing
by Peixin Qiao 15 Mar '17

15 Mar '17

Hi CODES, When I run the command in README.fattree.txt, I cannot finish creating static routing table. The third step cannot be finished as follows: peixin@peixin:~/simulation/scripts/routing_folder$ ls net.dot net.txt peixin@peixin:~/simulation/scripts/routing_folder$ cd .. peixin@peixin:~/simulation/scripts$ ~/simulation/scripts/create_static_lft.sh /home/peixin/simulation/scripts/routing_folder/ net dot file already exists. running createIBNet.py Selected configuration: Output: /home/peixin/simulation/scripts/routing_folder/topo.net Topology: XGFT Number of endpoints: 8 Number of ports per endpoints: 1 Number of switches: 4 Number of ports per switch: 36 Dimensions for Mesh/Torus: -1, -1, -1 k-ary-n Tree: -1 -ary- -1 Kautz K(b,n): K(-1; -1) XGFT(h,m,w): XGFT(3; 18,18,36; 1,18,18) Dragonfly(a,p,h) : Dragonfly(-1, -1, -1) with -1 groups Cascade(a,p,h,g) : Cascade(96, 8, 10) with -1 groups Tofu(d1,d2,d3) : Tofu(-1, -1, -1) with -1 groups This program also writes an dot-file to plot the graph with Graphviz. Try: dot / neato /home/peixin/simulation/scripts/routing_folder/topo.dot -Tpng -o graph.png && evince graph.png running simulate.py Run ibsim... Run opensm... ^CTraceback (most recent call last): File "/home/peixin/simulation/scripts/simulate.py", line 1140, in <module> app = runSim(options.network, options.routing, options.pattern) File "/home/peixin/simulation/scripts/simulate.py", line 826, in __init__ time.sleep(60) KeyboardInterrupt peixin@peixin:~/simulation/scripts$ cd routing_folder/ peixin@peixin:~/simulation/scripts/routing_folder$ ls log.txt net.dot net.txt ofedout rootguids.conf topo.dot topo.net peixin@peixin:~/simulation/scripts/routing_folder$ How to create fat-tree static routing table with 36-ports switch? Thanks, Peixin

2 5

Simulation ends with events unprocessed
by Jian Peng 06 Mar '17

06 Mar '17

Hi, I'm running some experiments on Dragonfly with CODES 5.0.2. Sometimes they end up with unsent messages remaining in terminals and routers. I think the reason is the events that carry these messages are not handled. I looked through my debugging information and confirmed it. So, why does ROSS end when there are still events unprocessed? I'm running a sequential simulation. Thanks. [image: Inline image 1]

3 5

CODES wiki
by Mubarak, Misbah 02 Mar '17

02 Mar '17

Hi All, Most of you might be already aware but we have a CODES wiki setup that provides details on the interconnect models, workload generators, best practices on model development, collective translation and much more. Please feel free to visit and share your feedback. https://xgitlab.cels.anl.gov/codes/codes/wikis/home Regards, Misbah

1 0

Questions about CODES
by Junji Kobayashi 28 Feb '17

28 Feb '17

Hi there, I want to simulate asynchronous RPC operation using CODES. (ex. LP x send a message to LP y and LP z using asynchronous RPC) Does CODES support asynchronous RPC? And how to simulate asynchronous RPC using CODES. Best Regards. Junji Kobayashi email:([email protected])

2 1

Fat-tree configuration support oversubscription >1?
by Peixin Qiao 01 Feb '17

01 Feb '17

Hi there, When I do research about fat-tree topology uisng CODES, the fat-tree.conf shows that the topology is with an oversubscription of 1:1, which indicates that all hosts may potentially communicate with arbitrary other hosts at the full bandwidth of their network interface. But the Stampede systems, which I simulated is with an oversubscription of 1.25:1, which means that each leaf switch with 36 ports is directly connected to 20 hosts and 16 ports in the core switch. So, does CODES support this configuration? And how should I modify the configuration file? Best Regards, Peixin Qiao

2 1