February 2014 - discuss - mailman3.cels.anl.gov

binary for win server 2012
by mcdougal 04 Feb '14

04 Feb '14

Hello, Is there an mpich2 binary package available for Windows Server 2012? Thanks, Bill

1 0

Re: [mpich-discuss] Parallel test hanging with mpich on rhel7
by Orion Poplawski 04 Feb '14

04 Feb '14

On 02/03/2014 07:29 PM, Orion Poplawski wrote: > We're starting to do the Fedora EPEL builds for EPEL7. I'm building > hdf5 1.8.12 with: > > mpich-3.0.4-4.el7.x86_64 > gcc-4.8.2-3.el7.x86_64 > > The following test hangs here: > > $ mpirun -np 4 ./t_cache > =================================== > Parallel metadata cache tests > mpi_size = 4 > express_test = 1 > =================================== > *** Hint *** > You can use environment variable HDF5_PARAPREFIX to run parallel test > files in a > different directory or to add file type prefix. E.g., > HDF5_PARAPREFIX=pfs:/PFS/user/me > export HDF5_PARAPREFIX > *** End of Hint *** > 0:setup_rand(): seed = 138071. > 3:setup_rand(): seed = 149196. > 2:setup_rand(): seed = 160135. > 1:setup_rand(): seed = 180134. > Testing server smoke check > PASSED > Testing smoke check #1 -- process 0 only md write strategy This turned out to be triggered when running oversubscribed. Reducing the number of mpi processes to equal or less than the number of cores made this particular hang go away. However, I'm still seeing a hang on our Fedora builders in a different test: make[4]: Entering directory `/builddir/build/BUILD/hdf5-1.8.12/mpich/testpar' ============================ Testing t_mpi Full log: http://koji.fedoraproject.org/koji/getfile?taskID=6492001&name=build.log Unfortunately I'm not able to reproduce this on my own machines so I'm at a loss here. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion(a)nwra.com Boulder, CO 80301 http://www.nwra.com

1 0

Re: [mpich-discuss] (no subject)
by Ricardo Román Brenes 04 Feb '14

04 Feb '14

Those xeon have hyper threading technology. They sell then as double the real core they have. On Feb 4, 2014 10:53 AM, "Reuti" <reuti(a)staff.uni-marburg.de> wrote: > Hi, > > Am 04.02.2014 um 17:45 schrieb Andreas Gross: > > > I am looking for help with MPICH2. > > > > I installed mpich-3.0.4 on a Centos 6.5 PC with 24 cores. > > When I launch with mpiexec on 17 procs all procs are running 100%. > > When I launch on more than 17 procs the job gets slower and most > processors run on less than 100%. > > > > I spent a lot of time google'ing this problem but did not find any > answers. > > Somebody said it may be a memory problem but the computer has 32GB RAM > and the same problem occurred when running a very small problem. > > On bigger computers I have run the same code on 2,048 processors with no > problem at all. > > > > > > > > "lscpu" gives the following response: > > > > Architecture: x86_64 > > CPU op-mode(s): 32-bit, 64-bit > > Byte Order: Little Endian > > CPU(s): 24 > > On-line CPU(s) list: 0-23 > > Thread(s) per core: 2 > > There are only 12 real core. Depending on the application it might degrade > when the threads are used. For our applications I found it might be around > factor 1.5 which can be used - this matches your detected limit of 17. > > -- Reuti > > > > Core(s) per socket: 6 > > Socket(s): 2 > > NUMA node(s): 2 > > Vendor ID: GenuineIntel > > CPU family: 6 > > Model: 45 > > Stepping: 7 > > CPU MHz: 1200.000 > > BogoMIPS: 4590.92 > > Virtualization: VT-x > > L1d cache: 32K > > L1i cache: 32K > > L2 cache: 256K > > L3 cache: 15360K > > NUMA node0 CPU(s): 0-5,12-17 > > NUMA node1 CPU(s): 6-11,18-23 > > > > > > > > "less /proc/cpuinfo" shows: > > > > processor : 0 > > vendor_id : GenuineIntel > > cpu family : 6 > > model : 45 > > model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz > > stepping : 7 > > cpu MHz : 1200.000 > > cache size : 15360 KB > > physical id : 0 > > siblings : 12 > > core id : 0 > > cpu cores : 6 > > apicid : 0 > > initial apicid : 0 > > fpu : yes > > fpu_exception : yes > > cpuid level : 13 > > wp : yes > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] (no subject)
by Reuti 04 Feb '14

04 Feb '14

Hi, Am 04.02.2014 um 17:45 schrieb Andreas Gross: > I am looking for help with MPICH2. > > I installed mpich-3.0.4 on a Centos 6.5 PC with 24 cores. > When I launch with mpiexec on 17 procs all procs are running 100%. > When I launch on more than 17 procs the job gets slower and most processors run on less than 100%. > > I spent a lot of time google'ing this problem but did not find any answers. > Somebody said it may be a memory problem but the computer has 32GB RAM and the same problem occurred when running a very small problem. > On bigger computers I have run the same code on 2,048 processors with no problem at all. > > > > "lscpu" gives the following response: > > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 24 > On-line CPU(s) list: 0-23 > Thread(s) per core: 2 There are only 12 real core. Depending on the application it might degrade when the threads are used. For our applications I found it might be around factor 1.5 which can be used - this matches your detected limit of 17. -- Reuti > Core(s) per socket: 6 > Socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 45 > Stepping: 7 > CPU MHz: 1200.000 > BogoMIPS: 4590.92 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 15360K > NUMA node0 CPU(s): 0-5,12-17 > NUMA node1 CPU(s): 6-11,18-23 > > > > "less /proc/cpuinfo" shows: > > processor : 0 > vendor_id : GenuineIntel > cpu family : 6 > model : 45 > model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz > stepping : 7 > cpu MHz : 1200.000 > cache size : 15360 KB > physical id : 0 > siblings : 12 > core id : 0 > cpu cores : 6 > apicid : 0 > initial apicid : 0 > fpu : yes > fpu_exception : yes > cpuid level : 13 > wp : yes > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

(no subject)
by Andreas Gross 04 Feb '14

04 Feb '14

I am looking for help with MPICH2. I installed mpich-3.0.4 on a Centos 6.5 PC with 24 cores. When I launch with mpiexec on 17 procs all procs are running 100%. When I launch on more than 17 procs the job gets slower and most processors run on less than 100%. I spent a lot of time google'ing this problem but did not find any answers. Somebody said it may be a memory problem but the computer has 32GB RAM and the same problem occurred when running a very small problem. On bigger computers I have run the same code on 2,048 processors with no problem at all. "lscpu" gives the following response: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 45 Stepping: 7 CPU MHz: 1200.000 BogoMIPS: 4590.92 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 15360K NUMA node0 CPU(s): 0-5,12-17 NUMA node1 CPU(s): 6-11,18-23 "less /proc/cpuinfo" shows: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 45 model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz stepping : 7 cpu MHz : 1200.000 cache size : 15360 KB physical id : 0 siblings : 12 core id : 0 cpu cores : 6 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes

1 0

Re: [mpich-discuss] Parallel test hanging with mpich on rhel7
by Orion Poplawski 03 Feb '14

03 Feb '14

On 02/03/2014 07:29 PM, Orion Poplawski wrote: > We're starting to do the Fedora EPEL builds for EPEL7. I'm building > hdf5 1.8.12 with: > > mpich-3.0.4-4.el7.x86_64 > gcc-4.8.2-3.el7.x86_64 > > The following test hangs here: > > $ mpirun -np 4 ./t_cache > =================================== > Parallel metadata cache tests > mpi_size = 4 > express_test = 1 > =================================== > *** Hint *** > You can use environment variable HDF5_PARAPREFIX to run parallel test > files in a > different directory or to add file type prefix. E.g., > HDF5_PARAPREFIX=pfs:/PFS/user/me > export HDF5_PARAPREFIX > *** End of Hint *** > 0:setup_rand(): seed = 138071. > 3:setup_rand(): seed = 149196. > 2:setup_rand(): seed = 160135. > 1:setup_rand(): seed = 180134. > Testing server smoke check > PASSED > Testing smoke check #1 -- process 0 only md write strategy > > Runs fine with openmpi. Not seeing problems either in Fedora, which has > similar versions, so really not sure what is at issue, or how to debug > further. I ran with: MPICH_DBG_LEVEL=VERBOSE MPICH_DBG=FILE log files are here: http://www.cora.nwra.com/~orion/hdf5-mpich-debug.tar.gz Hope that helps. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion(a)cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com

1 0

Parallel test hanging with mpich on rhel7
by Orion Poplawski 03 Feb '14

03 Feb '14

We're starting to do the Fedora EPEL builds for EPEL7. I'm building hdf5 1.8.12 with: mpich-3.0.4-4.el7.x86_64 gcc-4.8.2-3.el7.x86_64 The following test hangs here: $ mpirun -np 4 ./t_cache =================================== Parallel metadata cache tests mpi_size = 4 express_test = 1 =================================== *** Hint *** You can use environment variable HDF5_PARAPREFIX to run parallel test files in a different directory or to add file type prefix. E.g., HDF5_PARAPREFIX=pfs:/PFS/user/me export HDF5_PARAPREFIX *** End of Hint *** 0:setup_rand(): seed = 138071. 3:setup_rand(): seed = 149196. 2:setup_rand(): seed = 160135. 1:setup_rand(): seed = 180134. Testing server smoke check PASSED Testing smoke check #1 -- process 0 only md write strategy Runs fine with openmpi. Not seeing problems either in Fedora, which has similar versions, so really not sure what is at issue, or how to debug further. The processes are consuming 100% cpu - looping continually calling poll() (as seen by strace). Some gdb snapshot backtraces: (gdb) bt #0 0x00007fdfdcd9fae2 in MPIDI_CH3I_Progress () from /usr/lib64/mpich/lib/libmpich.so.10 #1 0x00007fdfdce495e8 in PMPI_Recv () from /usr/lib64/mpich/lib/libmpich.so.10 #2 0x0000000000403b24 in recv_mssg (mssg_ptr=0x7fffccf6e2c0, mssg_tag_offset=1) at ../../testpar/t_cache.c:1050 #3 0x000000000040655f in flush_datum (f=0x30fb600, dxpl_id=167772175, dest=0, addr=177681, thing=0x61cd10 <data+21840>) at ../../testpar/t_cache.c:2542 #4 0x00007fdfdd905513 in H5C_flush_single_entry (f=0x30fb600, primary_dxpl_id=167772175, secondary_dxpl_id=167772175, type_ptr=0x614d20 <types>, addr=177681, flags=0, first_flush_ptr=0x7fffccf6e464, del_entry_from_slist_on_destroy=0) at ../../src/H5C.c:7745 #5 0x00007fdfdd905f03 in H5C_make_space_in_cache (f=0x30fb600, primary_dxpl_id=167772175, secondary_dxpl_id=167772175, space_needed=0, write_permitted=1, first_flush_ptr=0x7fffccf6e464) at ../../src/H5C.c:8173 #6 0x00007fdfdd8f8a38 in H5C_flush_to_min_clean (f=0x30fb600, primary_dxpl_id=167772175, secondary_dxpl_id=167772175) at ../../src/H5C.c:1972 #7 0x00007fdfdd8d6298 in H5AC_rsp__p0_only__flush_to_min_clean (f=0x30fb600, dxpl_id=167772175, cache_ptr=0x7fdfddfb3010) at ../../src/H5AC.c:4754 #8 0x00007fdfdd8d64dc in H5AC_run_sync_point (f=0x30fb600, dxpl_id=167772175, sync_point_op=0) at ../../src/H5AC.c:4854 #9 0x00007fdfdd8ce3d3 in H5AC_insert_entry (f=0x30fb600, dxpl_id=167772168, type=0x614d20 <types>, addr=1856348, thing=0x64d360 <data+220064>, flags=0) at ../../src/H5AC.c:1006 #10 0x0000000000406cf9 in insert_entry (cache_ptr=0x7fdfddfb3010, file_ptr=0x30fb600, idx=1058, flags=0) at ../../testpar/t_cache.c:2939 #11 0x000000000040a542 in smoke_check_1 (metadata_write_strategy=0) at ../../testpar/t_cache.c:5375 #12 0x000000000040da3d in main (argc=1, argv=0x7fffccf6e798) at ../../testpar/t_cache.c:7261 (gdb) bt #0 0x00007ffd121dab2b in MPIDI_CH3I_Progress () from /usr/lib64/mpich/lib/libmpich.so.10 #1 0x00007ffd1218dd2d in MPIC_Wait () from /usr/lib64/mpich/lib/libmpich.so.10 #2 0x00007ffd1218e020 in MPIC_Recv () from /usr/lib64/mpich/lib/libmpich.so.10 #3 0x00007ffd1218e829 in MPIC_Recv_ft () from /usr/lib64/mpich/lib/libmpich.so.10 #4 0x00007ffd122131c2 in MPIR_Bcast_binomial.isra.1 () from /usr/lib64/mpich/lib/libmpich.so.10 #5 0x00007ffd122139da in MPIR_Bcast_intra () from /usr/lib64/mpich/lib/libmpich.so.10 #6 0x00007ffd1221428d in MPIR_Bcast_impl () from /usr/lib64/mpich/lib/libmpich.so.10 #7 0x00007ffd122147a2 in PMPI_Bcast () from /usr/lib64/mpich/lib/libmpich.so.10 #8 0x00007ffd12d0faba in H5AC_receive_and_apply_clean_list (f=0x3a535d0, primary_dxpl_id=167772175, secondary_dxpl_id=167772175, cache_ptr=0x7ffd133ee010) at ../../src/H5AC.c:4133 #9 0x00007ffd12d0f942 in H5AC_propagate_flushed_and_still_clean_entries_list (f=0x3a535d0, dxpl_id=167772175, cache_ptr=0x7ffd133ee010) at ../../src/H5AC.c:4073 #10 0x00007ffd12d11347 in H5AC_rsp__p0_only__flush_to_min_clean (f=0x3a535d0, dxpl_id=167772175, cache_ptr=0x7ffd133ee010) at ../../src/H5AC.c:4769 #11 0x00007ffd12d114dc in H5AC_run_sync_point (f=0x3a535d0, dxpl_id=167772175, sync_point_op=0) at ../../src/H5AC.c:4854 #12 0x00007ffd12d093d3 in H5AC_insert_entry (f=0x3a535d0, dxpl_id=167772168, type=0x614d20 <types>, addr=1856348, thing=0x64d360 <data+220064>, flags=0) at ../../src/H5AC.c:1006 #13 0x0000000000406cf9 in insert_entry (cache_ptr=0x7ffd133ee010, file_ptr=0x3a535d0, idx=1058, flags=0) at ../../testpar/t_cache.c:2939 #14 0x000000000040a542 in smoke_check_1 (metadata_write_strategy=0) at ../../testpar/t_cache.c:5375 #15 0x000000000040da3d in main (argc=1, argv=0x7fffe754ac68) at ../../testpar/t_cache.c:7261 (gdb) bt #0 0x00007f1aff58a2e0 in MPID_nem_tcp_connpoll () from /usr/lib64/mpich/lib/libmpich.so.10 #1 0x00007f1aff576b35 in MPIDI_CH3I_Progress () from /usr/lib64/mpich/lib/libmpich.so.10 #2 0x00007f1aff6205e8 in PMPI_Recv () from /usr/lib64/mpich/lib/libmpich.so.10 #3 0x0000000000403b24 in recv_mssg (mssg_ptr=0x7fff921b4210, mssg_tag_offset=11) at ../../testpar/t_cache.c:1050 #4 0x0000000000408b94 in verify_entry_writes (addr=250487, expected_entry_writes=1) at ../../testpar/t_cache.c:4506 #5 0x000000000040839b in verify_writes (num_writes=158, written_entries_tbl=0x282cda0) at ../../testpar/t_cache.c:4151 #6 0x00007f1b000abf5e in H5AC_receive_and_apply_clean_list (f=0x27fa5d0, primary_dxpl_id=167772175, secondary_dxpl_id=167772175, cache_ptr=0x7f1b0078a010) at ../../src/H5AC.c:4179 #7 0x00007f1b000ab942 in H5AC_propagate_flushed_and_still_clean_entries_list (f=0x27fa5d0, dxpl_id=167772175, cache_ptr=0x7f1b0078a010) at ../../src/H5AC.c:4073 #8 0x00007f1b000ad347 in H5AC_rsp__p0_only__flush_to_min_clean (f=0x27fa5d0, dxpl_id=167772175, cache_ptr=0x7f1b0078a010) at ../../src/H5AC.c:4769 #9 0x00007f1b000ad4dc in H5AC_run_sync_point (f=0x27fa5d0, dxpl_id=167772175, sync_point_op=0) at ../../src/H5AC.c:4854 #10 0x00007f1b000a53d3 in H5AC_insert_entry (f=0x27fa5d0, dxpl_id=167772168, type=0x614d20 <types>, addr=1856348, thing=0x64d360 <data+220064>, flags=0) at ../../src/H5AC.c:1006 #11 0x0000000000406cf9 in insert_entry (cache_ptr=0x7f1b0078a010, file_ptr=0x27fa5d0, idx=1058, flags=0) at ../../testpar/t_cache.c:2939 #12 0x000000000040a542 in smoke_check_1 (metadata_write_strategy=0) at ../../testpar/t_cache.c:5375 #13 0x000000000040da3d in main (argc=1, argv=0x7fff921b4668) at ../../testpar/t_cache.c:7261 Thanks for any help, Orion -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion(a)cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com

1 0

Re: [mpich-discuss] Help with Cray MPICH
by Marcin Zalewski 03 Feb '14

03 Feb '14

Thanks Nick, great to know. -m On Mon, Feb 3, 2014 at 2:46 PM, Nick Radcliffe <nradclif(a)cray.com> wrote: > Hi Marcin, > > I can't be certain, but this looks like a bug we fixed recently in Cray MPICH. The fix should be available in MPT 6.2.2 by Feb. 20. > > -Nick Radcliffe, > Cray MPT Team > > ________________________________________ > From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of Marcin Zalewski [marcin.zalewski(a)gmail.com] > Sent: Monday, February 03, 2014 1:27 PM > To: discuss(a)mpich.org > Subject: [mpich-discuss] Help with Cray MPICH > > I have an application I am trying to run on a Cray machine composed of > XE6 nodes. I have run this application previously using Open MPI and > MVAPICH on a few different machines, so I think it should be more or > less free of major bugs. However, when I run it on the Cray machine, I > get a segmentation fault with a stack trace that ends in this: > > #0 0x00002aaaafe265e6 in memcpy () from /lib64/libc.so.6 > #1 0x00002aaaae8fe023 in MPID_Segment_index_m2m () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #2 0x00002aaaae8fca18 in MPID_Segment_manipulate () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #3 0x00002aaaae9049d1 in MPID_Segment_unpack () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #4 0x00002aaaae8edd38 in MPID_nem_gni_complete_rdma_get () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #5 0x00002aaaae8e11c8 in MPID_nem_gni_check_localCQ () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #6 0x00002aaaae8e29fa in MPID_nem_gni_poll () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #7 0x00002aaaae8c3515 in MPIDI_CH3I_Progress () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #8 0x00002aaaae9a3d4d in PMPI_Testsome () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > > I understand that this is very little information to go on, but I > really cannot deal with this problem, and I thought I would try this > list while I am trying to solicit help from our support team. > Currently, I do not have access to a debug version of the library (or > the source code). Has anyone here seen a similar error? What would be > a reason to get an error on memcpy in MPICH while calling MPI_Testsome > in general? I do have asserts in my code to make sure that the output > arrays are large enough, and I got it to work on other implementations > of MPI, so I am really clueless to what the problem might be. > > Thank you, > Marcin > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Help with Cray MPICH
by Nick Radcliffe 03 Feb '14

03 Feb '14

Hi Marcin, I can't be certain, but this looks like a bug we fixed recently in Cray MPICH. The fix should be available in MPT 6.2.2 by Feb. 20. -Nick Radcliffe, Cray MPT Team ________________________________________ From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of Marcin Zalewski [marcin.zalewski(a)gmail.com] Sent: Monday, February 03, 2014 1:27 PM To: discuss(a)mpich.org Subject: [mpich-discuss] Help with Cray MPICH I have an application I am trying to run on a Cray machine composed of XE6 nodes. I have run this application previously using Open MPI and MVAPICH on a few different machines, so I think it should be more or less free of major bugs. However, when I run it on the Cray machine, I get a segmentation fault with a stack trace that ends in this: #0 0x00002aaaafe265e6 in memcpy () from /lib64/libc.so.6 #1 0x00002aaaae8fe023 in MPID_Segment_index_m2m () from /opt/cray/lib64/libmpich_gnu_48.so.2 #2 0x00002aaaae8fca18 in MPID_Segment_manipulate () from /opt/cray/lib64/libmpich_gnu_48.so.2 #3 0x00002aaaae9049d1 in MPID_Segment_unpack () from /opt/cray/lib64/libmpich_gnu_48.so.2 #4 0x00002aaaae8edd38 in MPID_nem_gni_complete_rdma_get () from /opt/cray/lib64/libmpich_gnu_48.so.2 #5 0x00002aaaae8e11c8 in MPID_nem_gni_check_localCQ () from /opt/cray/lib64/libmpich_gnu_48.so.2 #6 0x00002aaaae8e29fa in MPID_nem_gni_poll () from /opt/cray/lib64/libmpich_gnu_48.so.2 #7 0x00002aaaae8c3515 in MPIDI_CH3I_Progress () from /opt/cray/lib64/libmpich_gnu_48.so.2 #8 0x00002aaaae9a3d4d in PMPI_Testsome () from /opt/cray/lib64/libmpich_gnu_48.so.2 I understand that this is very little information to go on, but I really cannot deal with this problem, and I thought I would try this list while I am trying to solicit help from our support team. Currently, I do not have access to a debug version of the library (or the source code). Has anyone here seen a similar error? What would be a reason to get an error on memcpy in MPICH while calling MPI_Testsome in general? I do have asserts in my code to make sure that the output arrays are large enough, and I got it to work on other implementations of MPI, so I am really clueless to what the problem might be. Thank you, Marcin _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Help with Cray MPICH
by Marcin Zalewski 03 Feb '14

03 Feb '14

I have an application I am trying to run on a Cray machine composed of XE6 nodes. I have run this application previously using Open MPI and MVAPICH on a few different machines, so I think it should be more or less free of major bugs. However, when I run it on the Cray machine, I get a segmentation fault with a stack trace that ends in this: #0 0x00002aaaafe265e6 in memcpy () from /lib64/libc.so.6 #1 0x00002aaaae8fe023 in MPID_Segment_index_m2m () from /opt/cray/lib64/libmpich_gnu_48.so.2 #2 0x00002aaaae8fca18 in MPID_Segment_manipulate () from /opt/cray/lib64/libmpich_gnu_48.so.2 #3 0x00002aaaae9049d1 in MPID_Segment_unpack () from /opt/cray/lib64/libmpich_gnu_48.so.2 #4 0x00002aaaae8edd38 in MPID_nem_gni_complete_rdma_get () from /opt/cray/lib64/libmpich_gnu_48.so.2 #5 0x00002aaaae8e11c8 in MPID_nem_gni_check_localCQ () from /opt/cray/lib64/libmpich_gnu_48.so.2 #6 0x00002aaaae8e29fa in MPID_nem_gni_poll () from /opt/cray/lib64/libmpich_gnu_48.so.2 #7 0x00002aaaae8c3515 in MPIDI_CH3I_Progress () from /opt/cray/lib64/libmpich_gnu_48.so.2 #8 0x00002aaaae9a3d4d in PMPI_Testsome () from /opt/cray/lib64/libmpich_gnu_48.so.2 I understand that this is very little information to go on, but I really cannot deal with this problem, and I thought I would try this list while I am trying to solicit help from our support team. Currently, I do not have access to a debug version of the library (or the source code). Has anyone here seen a similar error? What would be a reason to get an error on memcpy in MPICH while calling MPI_Testsome in general? I do have asserts in my code to make sure that the output arrays are large enough, and I got it to work on other implementations of MPI, so I am really clueless to what the problem might be. Thank you, Marcin

1 0