Pavan,

1. Can you run your application processes using "ddd" or some other debugger to see where the double free is coming from? You might have to build mpich with --enable-g=dbg to get the debug symbols in.

Here is the full stack backtrace: ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- #0 0x00007ffff6deb475 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007ffff6dee6f0 in *__GI_abort () at abort.c:92 #2 0x00007ffff6e2652b in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189 #3 0x00007ffff6e2fd76 in malloc_printerr (action=3, str=0x7ffff6f081e0 "double free or corruption (fasttop)", ptr=<optimized out>) at malloc.c:6283 #4 0x00007ffff6e34aac in *__GI___libc_free (mem=<optimized out>) at malloc.c:3738 #5 0x00007ffff7a1d5d9 in populate_ids_from_mapping ( did_map=<synthetic pointer>, num_nodes=<synthetic pointer>, mapping=<optimized out>, pg=<optimized out>) at src/mpid/ch3/src/mpid_vc.c:1063 #6 MPIDI_Populate_vc_node_ids (pg=pg@entry=0x604910, our_pg_rank=our_pg_rank@entry=0) at src/mpid/ch3/src/mpid_vc.c:1193 #7 0x00007ffff7a17dd6 in MPID_Init (argc=argc@entry=0x7fffffffd97c, argv=argv@entry=0x7fffffffd970, requested=requested@entry=0, provided=provided@entry=0x7fffffffd8e8, has_args=has_args@entry=0x7fffffffd8e0, has_env=has_env@entry=0x7fffffffd8e4) at src/mpid/ch3/src/mpid_init.c:156 #8 0x00007ffff7acdf7f in MPIR_Init_thread (argc=argc@entry=0x7fffffffd97c, argv=argv@entry=0x7fffffffd970, required=required@entry=0, provided=provided@entry=0x7fffffffd944) at src/mpi/init/initthread.c:431 #9 0x00007ffff7acd90e in PMPI_Init (argc=0x7fffffffd97c, argv=0x7fffffffd970) at src/mpi/init/init.c:136 #10 0x000000000040086d in main () ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 -----

2. Can you send me the output with the ssh launcher as well?

See mail sent off-list. Thanks, Markus -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax: +49-2461-61-6656 E-mail: [email protected] WWW: http://www.fz-juelich.de/jsc/ ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------