- codes-ross-users - mailman3.cels.anl.gov

using OTF2 traces
by Sayan Ghosh 27 Sep '19

27 Sep '19

How do I use OTF2 traces generated from Score-P as a model workload type? I know DUMPI is the default, but I cannot seem to specify the other case. Also, I am passing the Score-P measurement directory with the trace files (one per process), hopefully that is the correct usage. model-net-mpi-replay --sync=1 --workload_type=<???> --workload_file=scorep-measurement-folder --num_net_traces=1024 -- modelnet-test-dragonfly.conf -- Thanks, Sayan | https://sg0.github.io/

1 1

Replaying OTF2 traces
by Ghosh, Sayan 25 Sep '19

25 Sep '19

Hi – I am trying to use CODES/ROSS to replay OTF2 traces (generated using ScoreP) on network models, after consulting the latest CODES/ROSS tutorial: https://www.mcs.anl.gov/projects/codes/files/2017/08/codes-hoti-tutorial-v4… I had used 1024 processes to generate OTF2 traces of my application (assuming it will run in serial mode without mpiexec). Ideally I would like to run the replay on my local machine using 1024 logical processes, without using mpiexec. Perhaps my understanding is not right here, please help. WE38523:codes-bw ghos167$ ~/builds/CODES/bin/model-net-mpi-replay --sync=3 --workload_type="online" --workload_file=scorep-measurement-tmp/ --num_net_traces=1024 -- ~/sources/CODES/tests/conf/modelnet-test-dragonfly.conf /Users/ghos167/builds/CODES/bin/model-net-mpi-replay --sync=3 --workload_type=online --workload_file=scorep-measurement-tmp/ --num_net_traces=1024 -- /Users/ghos167/sources/CODES/tests/conf/modelnet-test-dragonfly.conf Wed Sep 25 08:43:31 2019 ROSS Version: v7.1.1 tw_net_start: Found world size to be 1 Warning: Defaulting to Sequential Simulation, not enough PEs defined. NIC num injection port not specified, setting to 1 NIC seq delay not specified, setting to 10.000000 NIC num copy queues not specified, setting to 1 Total nodes 72 routers 36 groups 9 radix 8 within node transfer per byte delay is 0.190476 Within-node eager limit (node_eager_limit) not specified, setting to 16000 ROSS Core Configuration: Total PEs 1 Total KPs [Nodes (1) x KPs (16)] 16 Total LPs 180 Simulation End Time 3600000000000.00 LP-to-PE Mapping model defined ROSS Event Memory Allocation: Model events 46081 Network events 16 Total events 46096 Assertion failed: (num_net_traces <= num_mpi_lps), function nw_test_init, file src/network-workloads/model-net-mpi-replay.c, line 2107. Abort trap: 6 -- Sayan Data Sciences Pacific Northwest National Laboratory

1 0

CODES 1.1.0 installation issue
by Sayan Ghosh 29 Jul '19

29 Jul '19

I am reporting CODES build issue with Intel 19 and GCC 6.3, please suggest a workaround. I downloaded the software from: https://press3.mcs.anl.gov/codes/downloads/ .././src/util/codes_mapping.c(465): error: argument of type "tw_pe" is incompatible with parameter of type "tw_pe *" tw_kp_onpe(kpid, g_tw_pe[0]); ^ .././src/util/codes_mapping.c(477): warning #266: function "tw_getpe" declared implicitly pe = tw_getpe(kpid % g_tw_npe); ^ .././src/util/codes_mapping.c(477): error: identifier "g_tw_npe" is undefined pe = tw_getpe(kpid % g_tw_npe); ^ .././src/util/codes_mapping.c(477): warning #556: a value of type "int" cannot be assigned to an entity of type "tw_pe *" pe = tw_getpe(kpid % g_tw_npe); ^ -- Thanks, Sayan | https://sg0.github.io/

2 4

Hypercube
by Venkata Ayyalasomayajula 06 Jun '19

06 Jun '19

Dear Developers, Any near-future releases that might include an implementation of a Hypercube? Regards, Venkata

1 0

CODES version 1.1.0 released
by Mubarak, Misbah 16 Jan '19

16 Jan '19

Hi All, We are happy to announce the release of CODES version 1.1.0. This release comes after 6 months of the last release (1.0.0). The major updates and features include: * New Features: * Addition of a 1-D dragonfly network model in addition to the customized version of dragonfly network (dragonfly-dally.C). Multiple routing protocols including progressive adaptive routing and adaptive routing are supported. * Quality of Service support with 1-D dragonfly and megafly network models. More details about using QoS can be found on the wiki page<https://xgitlab.cels.anl.gov/codes/codes/wikis/Quality-of-Service>. * Addition of time-stepped series instrumentation data for the layer that simulates MPI operations and protocols. * Bug Fixes: * replace call for deprecated MPI_Type_hindexed * Fixed bug with mapping context which caused issues with simulating multiple ranks per node * Support for out of tree builds with megafly and 1-D dragonfly tests * Bug fix with on node and two level fat tree * Features in progress: * support for conceptual<https://ccsweb.lanl.gov/~pakin/software/conceptual/userguide/> workloads -- partial support is available in the repo. coNCePTuaL is a domain specific language/framework for auto-generating benchmarks to measure the performance and correctness of networks. Thanks, Misbah

1 0

Replaying darshan log in separate environment.
by kim 24 Sep '18

24 Sep '18

Hello. I am trying to replicate the application I/O characteristics from a server in another server by using a darshan log. I have installed CODES/ROSS and do have darshan log. Although i have been reading wikis, i cannot find information on how to replay the darshan log. Cods-io-workloads wiki does have some mention about darshan log but it is too abstract for me and i need some help. Can anyone help to be replay the darshan log or even guild to a page with more detailed information? Sorry that I'm new to CODES/ROSS and asking some basic question. Thank you in advance

2 1

Straggler event?
by Jian Peng 10 Aug '18

10 Aug '18

Hi All, Just ran into an issue with error: "error: network-mpi.c:388: 1:Received straggler from 7: 2938789.459012 3193751.109728 (0)" . Which is cause by failure of if(e->recv_ts < me->GVT) in recv_finish(tw_pe *me, tw_event *e, char * buffer) Any suggestion of fixing this issue? Thanks!

2 3

CODES Version 1.0.0 released!
by Mubarak, Misbah 16 Jul '18

16 Jul '18

Hi All, We are happy to announce the release of CODES version 1.0.0. This release comes after an year of the last release (0.6.0). The major updates and features include: * Adding support for dragonfly-plus network model. Multiple forms of routing (progressive adaptive, minimal, non-minimal-spine and leaf) have been implemented. https://xgitlab.cels.anl.gov/codes/codes/wikis/dragonfly-plus * Adding support for express mesh network model, which can be configured as hyperX. * Adding support for Multi-plane/rail in fat-tree via multiple single port NICs per compute node or one multi-port NIC per node. * Adding a generic template for building new network models. For simplest case, only 2 functions and premable changes should suffice to add a new network. Updated Express Mesh network model to serve as an example. For details, see src/networks/model-net/net-template.C * Darshan workload generator has been updated to use Darshan version 3.x. * Network models have been updated to capture simulation statistics over virtual time using ROSS/CODES instrumentation. For details, see: https://xgitlab.cels.anl.gov/codes/codes/wikis/Using-ROSS-Instrumentation-w… * Compatible with ROSS version that enables statistics collection of simulation performance. For details see: http://carothersc.github.io/ROSS/instrumentation/instrumentation.html * Online workload replay functionality has been added that allows SWM workloads to be simulated insitu on the network models. WIP to integrate Conceptual domain specific language for network communication. * Multiple traffic patterns were added in the background traffic generation including stencil, all-to-all and random permutation. * Performance tuning enabled for optimistic mode. For details, see: https://xgitlab.cels.anl.gov/codes/codes/wikis/Optimistic-Performance-Tunin… The release is available for download here<ftp://ftp.mcs.anl.gov/pub/CODES/releases/codes-1.0.0.tar.gz>. http://press3.mcs.anl.gov/codes/2018/07/13/codes-version-1-0-0-released/ Thanks, Misbah

1 0

ROSS Optimistic details and tips
by Caitlin Ross 02 Jul '18

02 Jul '18

Hi all, Just wanted to send this out to let people know that we now have some more details online on the parameters that can be used for tuning ROSS optimistic mode: http://carothersc.github.io/ROSS/feature/optimistic-params.html <http://carothersc.github.io/ROSS/feature/optimistic-params.html> I also added a page to the CODES wiki to give some tips on settings for these parameters for CODES models: https://xgitlab.cels.anl.gov/codes/codes/wikis/Optimistic-Performance-Tunin… <https://xgitlab.cels.anl.gov/codes/codes/wikis/Optimistic-Performance-Tunin…> If you don’t know much about the various parameters that can be used to tune optimistic performance, I recommend checking these out. Also, if you have found success with some settings that aren’t listed on the CODES wiki, feel free to add them (or let me know, and I can add them if you don’t have access to make changes to the wiki). Thanks, Caitlin

2 1

Load imbalance when running TraceR in parallel
by Taffet, Philip Adam 02 Jul '18

02 Jul '18

Hi, I’m trying to run an TraceR OTF simulation with lots of messages and lots of congestion. This is the first time that I’ve had a big enough simulation that I need to run it in parallel, and I’m having a really hard time getting any sort of parallel speedup. I tried running on 4-8 nodes with –sync=2 and –sync=3, as well as various values of --nkp, and the best I’ve gotten is only a few percent faster than serial. I looked into it some, and found that the cause appears to be massive load imbalance. I’m attaching a screenshot from hpctraceviewer that shows that rank 0 does almost all the work while the other ranks spend a large amount of time in MPI_Allreduce, waiting for rank 0 to arrive. I don’t know this part of ROSS/CODES very well, but does this mean the LPs are not being distributed evenly? If so, how can I change the distribution? It wouldn’t surprise me if my traffic pattern caused some load imbalance because there are 4 endpoints that receive way more traffic than the others, but I don’t think the imbalance should be this bad. Thank you very much, Philip Taffet

2 1