Great! We opened up a gitlab issue to see if we can add a graceful safety check for that case. thanks, -Phil On 05/24/2016 05:30 PM, Salmon, Rene wrote:

Success at last. Thank you all for the help. Again the info you provided gave me enough hints. For whatever reason the version of gnuplot on my system did not know what to do with the �-e� flag. All I did was upgrade gnuplot and now I have a nice PDF with graphs.

This is good enough for me for now. I will run a few test and see what I can learn from the tool.

Thanks again Rene

On 5/24/16, 10:28 AM, "Phil Carns" <[email protected]> wrote:

...

Gnuplot 4.2 should be Ok. I just confirmed that 4.2 patchlevel 6 works with darshan-job-summary.pl on one of our systems.

You could try the following to check what happens to gnuplot when it tries to generate the first plot:

- add --verbose to the darshan-job-summary.pl command line - look for "verbose: /tmp/1FMc4r0alx" (or something similar) in the first line of output - cd to that directory - run the gnuplot command manually:

gnuplot -e "data_file='mpiio-access-hist.dat'; graph_title='MPI-IO Access Sizes {/Times-Bold=32 \263}'; output_file='mpiio-access-hist.eps'" access-hist-eps.gplt

In theory it should run cleanly and create a mpiio-access-hist.eps output file.

Thanks for your patience getting this stuff working, by the way. I'm not sure what can be done about the Intel MPI Fortran wrapper problem on our end, but if there is a gnuplot issue we should be able to work around it or at least gracefully detect the problem.

thanks, -Phil

On 05/24/2016 10:37 AM, Salmon, Rene wrote:

...

Hi,

hpcbuild02(salmr0)1060:gnuplot -V gnuplot 4.2 patchlevel 3

Thanks Rene

On 5/24/16, 9:35 AM, "Phil Carns" <[email protected]> wrote:

...

Hi Rene,

What version of gnuplot do you have ("gnuplot -V")?

Darshan is supposed to have a safety check for the presence of gnuplot and it's version, but the "-e" thing is part of the gnuplot command line. Maybe something has gone wrong there.

thanks, -Phil

On 05/24/2016 10:21 AM, Salmon, Rene wrote:

...

All,

Thank you very much for all the help and info. I now have a working binary that does generate a darshan log file. Your hints where very helpful. What I ended up doing was writing an mpi_init and mpi_finalize wrapper in C that my fortran code calls and that seems to work and now I can generate a darshan log file.

I basically just call my_finalize() from my fortran code and that is just this:

#include "mpi.h" int my_finalize_(void) { int ier; ier = MPI_Finalize(); return(ier); }

Now I just want to analyze the results so I run the darshan-job-summary.pl but I am getting these errors. Any ideas what they mean?

hpcbuild02(salmr0)1054:darshan-job-summary.pl ./salmr0_ddswriter_id40315_5-24-32872-5756027388706526063_1.darshan Slowest unique file time: 0.004726 Slowest shared file time: 0.011293 Total bytes read and written by app (may be incorrect): 51934 Total absolute I/O time: 0.016019 **NOTE: above shared and unique file times calculated using MPI-IO timers if MPI-IO interface used on a given file, POSIX timers otherwise. Cannot open load file '-e' line 0: util.c: No such file or directory

EPSTOPDF 2.9.5gw, 2006/01/29 - Copyright 1998-2006 by Sebastian Rahtz et al. !!! Error: 'posix-access-hist.eps' does not exist! Cannot open load file '-e' line 0: util.c: No such file or directory

EPSTOPDF 2.9.5gw, 2006/01/29 - Copyright 1998-2006 by Sebastian Rahtz et al. !!! Error: 'file-access-read.eps' does not exist! EPSTOPDF 2.9.5gw, 2006/01/29 - Copyright 1998-2006 by Sebastian Rahtz et al. !!! Error: 'file-access-write.eps' does not exist! EPSTOPDF 2.9.5gw, 2006/01/29 - Copyright 1998-2006 by Sebastian Rahtz et al. !!! Error: 'file-access-shared.eps' does not exist! LaTeX generation (phase1) failed [17920], aborting summary creation. error log: Overfull \hbox (18.76118pt too wide) in paragraph at lines 52--52 [][] <time-summary.pdf, id=1, 289.08pt x 252.945pt> <use time-summary.pdf> <op-counts.pdf, id=2, 289.08pt x 252.945pt> <use op-counts.pdf>

LaTeX Warning: File `posix-access-hist.pdf' not found on input line 68.

!pdfTeX error: pdflatex (file posix-access-hist.pdf): cannot find image file ==> Fatal error occurred, no output PDF file produced!

Thanks Rene

On 5/23/16, 5:21 PM, "Harms, Kevin" <[email protected]> wrote:

...

Rene,

Your code is making the mpi_init and mpi_finalize call from Fortran correct? You are still hitting the same issue as you noted before which is that the Darshan doesn�t intercept those calls with the Intel MPI implementation. You could specifically add the darshan calls to your Fortran code manually. It isn�t pretty but it should work. This will initialize the library and then it should capture all the calls from your I/O code since that is all in C.

call mpi_init() call darshan_core_initialize(%VAL(0), %VAL(0)); � call darshan_core_shutdown(); call mpi_finalize()

depending on the function names, you may need to add an underscore to the end of the names.

kevin

...

Hi Phil,

Thanks for the info that was helpful. I did manage to link in darshan and MPI statically I had to put in the �lib.a� files by hand as the �darshan-config �pre/post� scripts where still trying to link in darshan dynamically. Here is the link line I used that seem to generate a binary.

mpiifort -DLinux -Dx86_64 -DINTEL -DCompLevel16 -DX86_64 -D__x86_64__ -static-intel -static_mpi -DOpenMP -openmp -DSWAP -O3 -g -o /hpcdata/salmr0/mspain/usp2hdf5/dds/bin/Linux/3.0/x86_64/ddswriter.revUnversioned /hpc/tstapps/intel/x86_64/ics-2016.update.1/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib/libmpifort.a -L/hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/lib /hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/lib/libdarshan.a -lz -Wl,@/hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/share/ld-opts/darshan-base-ld-opts svn_version.o PartitionLayout_mod.o ddswriter.o -L/tstapps/asi/src/dds/lib/Linux/3.0/x86_64/Intel-16.0.1.150MPI5.1.2MKL11.3.1 -ldds_r3 -lgio -lfhost -L/hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/lib -Wl,--start-group /hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/lib/libdarshan.a /hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/lib/libdarshan-stubs.a -Wl,--end-group -lz

Here is what the binary look like:

hpcbuild02(salmr0)1181:ldd bin/Linux/3.0/x86_64/ddswriter linux-vdso.so.1 => (0x00007ffe791a7000) libz.so.1 => /lib64/libz.so.1 (0x00002aafaab7c000) libdl.so.2 => /lib64/libdl.so.2 (0x00002aafaad92000) librt.so.1 => /lib64/librt.so.1 (0x00002aafaaf96000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aafab1a0000) libm.so.6 => /lib64/libm.so.6 (0x00002aafab3bd000) libc.so.6 => /lib64/libc.so.6 (0x00002aafab636000) /lib64/ld-linux-x86-64.so.2 (0x00002aafaa95a000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aafab9b3000) hpcbuild02(salmr0)1182:nm bin/Linux/3.0/x86_64/ddswriter|grep darshan 0000000000482e90 T darshan_clean_file_path 0000000000483210 t darshan_common_val_compare 00000000004830c0 T darshan_common_val_counter 00000000004831a0 t darshan_common_val_walker 000000000101e860 b darshan_core 0000000000463f50 t darshan_core_cleanup 00000000004655e0 T darshan_core_initialize 0000000000fcfa80 d darshan_core_mutex 0000000000464720 T darshan_core_register_module 0000000000464b80 T darshan_core_register_record 0000000000461c10 T darshan_core_shutdown 0000000000464840 T darshan_core_unregister_module 0000000000465310 T darshan_core_unregister_record 00000000004655b0 T darshan_core_wtime 0000000000463e40 t darshan_deflate_buffer..0 00000000004642b0 t darshan_get_shared_records 00000000004806a0 T darshan_hash 0000000000fcfab0 d darshan_mem_alignment 0000000000fcfaec d darshan_mem_alignment 0000000000b64848 r darshan_module_names 0000000000b6487c r darshan_module_versions 0000000000479ae0 T darshan_mpiio_shutdown_bench_setup 0000000000fcfa20 D darshan_path_exclusions 0000000000470250 T darshan_posix_shutdown_bench_setup 0000000000461750 T darshan_shutdown_bench 0000000000483230 T darshan_variance_reduce 0000000000483170 T darshan_walk_common_vals

The problem I am having now is that I don�t get a darshan log file when I run my executable. I configured darshan with this configure flag �--with-log-path-by-env=DARSHANLOGPATH�. I set that environment variable in my shell to some directory and then run my executable. The executable runs fine but no darshan log file gets generated.

Any ideas? Thanks in advanced.

Rene

On 5/20/16, 7:33 AM, "[email protected] on behalf of Phil Carns" <[email protected] on behalf of [email protected]> wrote:

...

Hi Rene,

Does mpiifort support the "-show" option (or something similar) to display the underlying link command that it is using?

I can describe the manual steps needed to add Darshan support just to see if you can get it working, and then we can maybe circle back to why the Darshan-generated script is triggering dynamic linking. In general, there are two sets of extra link flags that Darshan needs to add to the link command. Some should appear before the user-specified libraries in the true link command, some should appear after them. On my test box they happen to look like this:

pcarns@carns-x1:~/working/install/bin$ darshan-config --pre-ld-flags -L/home/pcarns/working/install/lib -ldarshan -lz -Wl,@/home/pcarns/working/install/share/ld-opts/darshan-base-ld-opts

pcarns@carns-x1:~/working/install/bin$ darshan-config --post-ld-flags -L/home/pcarns/working/install/lib -Wl,--start-group -ldarshan -ldarshan-stubs -Wl,--end-group -lz -lrt -lpthread

On most MPI implementations there is also one more library that you have to add for MPI function wrapping to work. Darshan-config doesn't add that because it is dependent on the MPI implementation. On most versions of MPI this is either -lmpifort or -lfmpich. I'm not sure which one Intel MPI uses now.

You might be able to do all of that external to mpiifort, but I'm not sure. If so, then your link command would look something like this (possibly swapping out -lmpifort for -lfmpich or omitting it entirely):

mpiifort -DLinux -Dx86_64 -DINTEL -DCompLevel16 -DX86_64 -D__x86_64__ -static-intel -static_mpi -DOpenMP -openmp -DSWAP -O3 -g -o ./ddswriter.revUnversioned -lmpifort `darshan-config --pre-ld-flags` PartitionLayout_mod.o ddswriter.o -L/tstapps/asi/src/dds/lib/Linux/3.0/x86_64/Intel-16.0.1.150MPI5.1.2MKL11.3.1 -ldds_r3 -lgio -lfhost `darshan-config --post-ld-flags`

If that doesn't work then you'll need to find out what the underlying link command looks like (via -show or similar) and insert it there.

thanks, -Phil

_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users