July 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_Comm_disconnect and pending communications
by Seo, Sangmin 26 Jul '16

26 Jul '16

Hi Jorge, You can try the debug event logging described in https://wiki.mpich.org/mpich/index.php/Debug_Event_Logging. Regards, Sangmin On Jul 26, 2016, at 11:57 AM, Jorge Bellón <jorge.bellon(a)bsc.es<mailto:[email protected]>> wrote: Hi, I am currently working with dynamic process creation with MPI using MPI_Comm_spawn_multiple. However, I'm facing issues during the application cleanup: MPI_Comm_disconnect function is hung and does not ever return (it is being called by all the processes involved in the intercommunicator). I know this issue could happen when there are unfinished communications (this has already happened to me in the past), in a similar way MPI_Finalize waits until all the communications in all the communicators are completed. This time, though, I'm not able to track which communications haven't finished yet. I was able to trace all the calls to the library using gdb and breakpoints, but everything seems OK to me. Is there any way to log the start and arrival of the messages (maybe using configure "--enable-g" option or setting one of the environment variables)? Thanks in advance. Regards, Jorge WARNING / LEGAL TEXT: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. http://www.bsc.es/disclaimer _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error while using MPI_Send
by Doha Ehab 26 Jul '16

26 Jul '16

changing the hostname and adding it along with IP address in /etc/hosts made it work. Thanks Doha On Mon, Jul 25, 2016 at 8:41 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > This sometimes happens if the node cannot resolve its hostname itself. > Can you try adding your hostname and IP to /etc/hosts? > > - Pavan > > > On Jul 25, 2016, at 1:36 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> > wrote: > > > > Hi, > > > > Normally this type of error would be the result of a firewall blocking > communication, but your run is on a single node so that shouldn't be the > case. I wonder, since it did work at one point, if there is some bad state > on your system that might be cleared with a reboot? > > > > It also looks like you are running on a Android system, which we do not > have much experience with, and no good way to test ourselves so our ways to > help may be limited. > > > > Ken > > > > On 07/24/2016 11:50 AM, Doha Ehab wrote: > >> Hello > >> I am using a cross compiled version of MPICH3, I was trying simple > >> containing MPI_Send and MPI_Recv, it was working but suddenly I keep > >> receiving this error messsage: can anyone point out what is wrong and > >> how to fix it. > >> > >> $ mpiexec -v -n 4 /data/parallelCode > >> host: tab > >> > >> > ================================================================================================== > >> mpiexec options: > >> ---------------- > >> Base path: /system/xbin/ > >> Launcher: (null) > >> Debug level: 1 > >> Enable X: -1 > >> > >> Global environment: > >> ------------------- > >> _=/system/xbin/mpiexec > >> PATH=/sbin:/vendor/bin:/system/sbin:/system/bin:/system/xbin > >> LOOP_MOUNTPOINT=/mnt/obb > >> ANDROID_ROOT=/system > >> SHELL=/system/bin/sh > >> ANDROID_DATA=/data > >> ANDROID_ASSETS=/system/app > >> TERM=vt100 > >> ANDROID_PROPERTY_WORKSPACE=8,0 > >> ANDROID_BOOTLOGO=1 > >> HOSTNAME=hwt1701 > >> LD_LIBRARY_PATH=/vendor/lib:/system/lib > >> > >> > BOOTCLASSPATH=/system/framework/core.jar:/system/framework/conscrypt.jar:/system/framework/okhttp.jar:/system/framework/core-junit.jar:/system/framework/bouncycastle.jar:/system/framework/ext.jar:/system/framework/framework.jar:/system/framework/framework2.jar:/system/framework/hwframework.jar:/system/framework/hwcustframework.jar:/system/framework/telephony-common.jar:/system/framework/voip-common.jar:/system/framework/mms-common.jar:/system/framework/android.policy.jar:/system/framework/services.jar:/system/framework/apache-xml.jar:/system/framework/webviewchromium.jar:/system/framework/hwEmui.jar:/system/framework/hwServices.jar:/system/framework/hwAndroid.policy.jar:/system/framework/hwTelephony-common.jar:/system/framework/hwpadext.jar > >> EMULATED_STORAGE_SOURCE=/mnt/shell/emulated > >> ANDROID_SOCKET_adbd=10 > >> EMULATED_STORAGE_TARGET=/storage/emulated > >> ANDROID_STORAGE=/storage > >> MKSH=/system/bin/sh > >> EXTERNAL_STORAGE=/storage/emulated/legacy > >> USBHOST_STORAGE=/storage/usbdisk > >> RANDOM=11338 > >> ASEC_MOUNTPOINT=/mnt/asec > >> SECONDARY_STORAGE=/storage/sdcard1 > >> USER=shell > >> LEGACY_STORAGE=/storage/emulated/legacy > >> HOME=/data > >> > >> Hydra internal environment: > >> --------------------------- > >> GFORTRAN_UNBUFFERED_PRECONNECTED=y > >> > >> > >> Proxy information: > >> ********************* > >> [1] proxy: tab (1 cores) > >> Exec list: /data/mmp100 (4 processes); > >> > >> > >> > ================================================================================================== > >> > >> [mpiexec@tab] Timeout set to -1 (-1 means infinite) > >> [mpiexec@tab] Got a control port string of tab:48661 > >> > >> Proxy launch args: /system/xbin/hydra_pmi_proxy --control-port tab:48661 > >> --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 > >> --usize -2 --proxy-id > >> > >> Arguments being passed to proxy 0: > >> --version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME > >> --hostname tab --global-core-map 0,1,1 --pmi-id-map 0,0 > >> --global-process-count 4 --auto-cleanup 1 --pmi-kvsname kvs_10003_0 > >> --pmi-process-mapping (vector,(0,1,1)) --ckpoint-num -1 > >> --global-inherited-env 26 '_=/system/xbin/mpiexec' > >> 'PATH=/sbin:/vendor/bin:/system/sbin:/system/bin:/system/xbin' > >> 'LOOP_MOUNTPOINT=/mnt/obb' 'ANDROID_ROOT=/system' 'SHELL=/system/bin/sh' > >> 'ANDROID_DATA=/data' 'ANDROID_ASSETS=/system/app' 'TERM=vt100' > >> 'ANDROID_PROPERTY_WORKSPACE=8,0' 'ANDROID_BOOTLOGO=1' 'HOSTNAME=hwt1701' > >> 'LD_LIBRARY_PATH=/vendor/lib:/system/lib' > >> > 'BOOTCLASSPATH=/system/framework/core.jar:/system/framework/conscrypt.jar:/system/framework/okhttp.jar:/system/framework/core-junit.jar:/system/framework/bouncycastle.jar:/system/framework/ext.jar:/system/framework/framework.jar:/system/framework/framework2.jar:/system/framework/hwframework.jar:/system/framework/hwcustframework.jar:/system/framework/telephony-common.jar:/system/framework/voip-common.jar:/system/framework/mms-common.jar:/system/framework/android.policy.jar:/system/framework/services.jar:/system/framework/apache-xml.jar:/system/framework/webviewchromium.jar:/system/framework/hwEmui.jar:/system/framework/hwServices.jar:/system/framework/hwAndroid.policy.jar:/system/framework/hwTelephony-common.jar:/system/framework/hwpadext.jar' > >> 'EMULATED_STORAGE_SOURCE=/mnt/shell/emulated' 'ANDROID_SOCKET_adbd=10' > >> 'EMULATED_STORAGE_TARGET=/storage/emulated' 'ANDROID_STORAGE=/storage' > >> 'MKSH=/system/bin/sh' 'EXTERNAL_STORAGE=/storage/emulated/legacy' > >> 'USBHOST_STORAGE=/storage/usbdisk' 'RANDOM=11338' > >> 'ASEC_MOUNTPOINT=/mnt/asec' 'SECONDARY_STORAGE=/storage/sdcard1' > >> 'USER=shell' 'LEGACY_STORAGE=/storage/emulated/legacy' 'HOME=/data' > >> --global-user-env 0 --global-system-env 1 > >> 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec > >> --exec-appnum 0 --exec-proc-count 4 --exec-local-env 0 --exec-wdir / > >> --exec-args 1 /data/mmp100 > >> > >> [mpiexec@tab] Launch arguments: /system/xbin/hydra_pmi_proxy > >> --control-port tab:48661 --debug --rmk user --launcher ssh --demux poll > >> --pgid 0 --retries 10 --usize -2 --proxy-id 0 > >> [proxy:0:0@tab] got pmi command (from 0): init > >> pmi_version=1 pmi_subversion=1 > >> [proxy:0:0@tab] PMI response: cmd=response_to_init pmi_version=1 > >> pmi_subversion=1 rc=0 > >> [proxy:0:0@tab] got pmi command (from 0): get_maxes > >> > >> [proxy:0:0@tab] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 > >> vallen_max=1024 > >> [proxy:0:0@tab] got pmi command (from 6): init > >> pmi_version=1 pmi_subversion=1 > >> [proxy:0:0@tab] PMI response: cmd=response_to_init pmi_version=1 > >> pmi_subversion=1 rc=0 > >> [proxy:0:0@tab] got pmi command (from 6): get_maxes > >> > >> [proxy:0:0@tab] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 > >> vallen_max=1024 > >> [proxy:0:0@tab] got pmi command (from 9): init > >> pmi_version=1 pmi_subversion=1 > >> [proxy:0:0@tab] PMI response: cmd=response_to_init pmi_version=1 > >> pmi_subversion=1 rc=0 > >> [proxy:0:0@tab] got pmi command (from 15): init > >> pmi_version=1 pmi_subversion=1 > >> [proxy:0:0@tab] PMI response: cmd=response_to_init pmi_version=1 > >> pmi_subversion=1 rc=0 > >> [proxy:0:0@tab] got pmi command (from 0): get_appnum > >> > >> [proxy:0:0@tab] PMI response: cmd=appnum appnum=0 > >> [proxy:0:0@tab] got pmi command (from 9): get_maxes > >> > >> [proxy:0:0@tab] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 > >> vallen_max=1024 > >> [proxy:0:0@tab] got pmi command (from 0): get_my_kvsname > >> > >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 > >> [proxy:0:0@tab] got pmi command (from 15): get_maxes > >> > >> [proxy:0:0@tab] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 > >> vallen_max=1024 > >> [proxy:0:0@tab] got pmi command (from 0): get_my_kvsname > >> > >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 > >> [proxy:0:0@tab] got pmi command (from 9): get_appnum > >> > >> [proxy:0:0@tab] PMI response: cmd=appnum appnum=0 > >> [proxy:0:0@tab] got pmi command (from 0): get > >> kvsname=kvs_10003_0 key=PMI_process_mapping > >> [proxy:0:0@tab] PMI response: cmd=get_result rc=0 msg=success > >> value=(vector,(0,1,1)) > >> [proxy:0:0@tab] got pmi command (from 15): get_appnum > >> > >> [proxy:0:0@tab] PMI response: cmd=appnum appnum=0 > >> [proxy:0:0@tab] got pmi command (from 6): get_appnum > >> > >> [proxy:0:0@tab] PMI response: cmd=appnum appnum=0 > >> [proxy:0:0@tab] got pmi command (from 9): get_my_kvsname > >> > >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 > >> [proxy:0:0@tab] got pmi command (from 15): get_my_kvsname > >> > >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 > >> [proxy:0:0@tab] got pmi command (from 6): get_my_kvsname > >> > >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 > >> [proxy:0:0@tab] got pmi command (from 6): get_my_kvsname > >> > >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 > >> [proxy:0:0@tab] got pmi command (from 9): get_my_kvsname > >> > >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 > >> [proxy:0:0@tab] got pmi command (from 0): put > >> kvsname=kvs_10003_0 key=P0-businesscard > >> value=port#49751$description#tab$ifname#192.168.1.4$ > >> [proxy:0:0@tab] cached command: > >> P0-businesscard=port#49751$description#tab$ifname#192.168.1.4$ > >> [proxy:0:0@tab] PMI response: cmd=put_result rc=0 msg=success > >> [proxy:0:0@tab] got pmi command (from 9): get > >> kvsname=kvs_10003_0 key=PMI_process_mapping > >> [proxy:0:0@tab] PMI response: cmd=get_result rc=0 msg=success > >> value=(vector,(0,1,1)) > >> [proxy:0:0@tab] got pmi command (from 0): barrier_in > >> > >> [proxy:0:0@tab] got pmi command (from 6): get > >> kvsname=kvs_10003_0 key=PMI_process_mapping > >> [proxy:0:0@tab] PMI response: cmd=get_result rc=0 msg=success > >> value=(vector,(0,1,1)) > >> [proxy:0:0@tab] got pmi command (from 15): get_my_kvsname > >> > >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 > >> [proxy:0:0@tab] got pmi command (from 9): put > >> kvsname=kvs_10003_0 key=P2-businesscard > >> value=port#60729$description#tab$ifname#192.168.1.4$ > >> [proxy:0:0@tab] cached command: > >> P2-businesscard=port#60729$description#tab$ifname#192.168.1.4$ > >> [proxy:0:0@tab] PMI response: cmd=put_result rc=0 msg=success > >> [proxy:0:0@tab] got pmi command (from 15): get > >> kvsname=kvs_10003_0 key=PMI_process_mapping > >> [proxy:0:0@tab] PMI response: cmd=get_result rc=0 msg=success > >> value=(vector,(0,1,1)) > >> [proxy:0:0@tab] got pmi command (from 9): barrier_in > >> > >> [proxy:0:0@tab] got pmi command (from 6): put > >> kvsname=kvs_10003_0 key=P1-businesscard > >> value=port#44344$description#tab$ifname#192.168.1.4$ > >> [proxy:0:0@tab] cached command: > >> P1-businesscard=port#44344$description#tab$ifname#192.168.1.4$ > >> [proxy:0:0@tab] PMI response: cmd=put_result rc=0 msg=success > >> [proxy:0:0@tab] got pmi command (from 15): put > >> kvsname=kvs_10003_0 key=P3-businesscard > >> value=port#51326$description#tab$ifname#192.168.1.4$ > >> [proxy:0:0@tab] cached command: > >> P3-businesscard=port#51326$description#tab$ifname#192.168.1.4$ > >> [proxy:0:0@tab] PMI response: cmd=put_result rc=0 msg=success > >> [proxy:0:0@tab] got pmi command (from 6): barrier_in > >> > >> [proxy:0:0@tab] got pmi command (from 15): barrier_in > >> > >> [proxy:0:0@tab] flushing 4 put command(s) out > >> [mpiexec@tab] [pgid: 0] got PMI command: cmd=put > >> P0-businesscard=port#49751$description#tab$ifname#192.168.1.4$ > >> P2-businesscard=port#60729$description#tab$ifname#192.168.1.4$ > >> P1-businesscard=port#44344$description#tab$ifname#192.168.1.4$ > >> P3-businesscard=port#51326$description#tab$ifname#192.168.1.4$ > >> [proxy:0:0@tab] forwarding command (cmd=put > >> P0-businesscard=port#49751$description#tab$ifname#192.168.1.4$ > >> P2-businesscard=port#60729$description#tab$ifname#192.168.1.4$ > >> P1-businesscard=port#44344$description#tab$ifname#192.168.1.4$ > >> P3-businesscard=port#51326$description#tab$ifname#192.168.1.4$) upstream > >> [proxy:0:0@tab] forwarding command (cmd=barrier_in) upstream > >> [mpiexec@tab] [pgid: 0] got PMI command: cmd=barrier_in > >> [mpiexec@tab] PMI response to fd 6 pid 15: cmd=keyval_cache > >> P0-businesscard=port#49751$description#tab$ifname#192.168.1.4$ > >> P2-businesscard=port#60729$description#tab$ifname#192.168.1.4$ > >> P1-businesscard=port#44344$description#tab$ifname#192.168.1.4$ > >> P3-businesscard=port#51326$description#tab$ifname#192.168.1.4$ > >> [mpiexec@tab] PMI response to fd 6 pid 15: cmd=barrier_out > >> [proxy:0:0@tab] PMI response: cmd=barrier_out > >> [proxy:0:0@tab] PMI response: cmd=barrier_out > >> [proxy:0:0@tab] PMI response: cmd=barrier_out > >> [proxy:0:0@tab] PMI response: cmd=barrier_out > >> [proxy:0:0@tab] got pmi command (from 0): get > >> kvsname=kvs_10003_0 key=P1-businesscard > >> [proxy:0:0@tab] PMI response: cmd=get_result rc=0 msg=success > >> value=port#44344$description#tab$ifname#192.168.1.4$ > >> Fatal error in MPI_Send: Unknown error class, error stack: > >> MPI_Send(174)...............................: MPI_Send(buf=0x15c56c, > >> count=1, MPI_INT, dest=1, tag=1, MPI_COMM_WORLD) failed > >> MPIDI_CH3i_Progress_wait(242)...............: an error occurred while > >> handling an event returned by MPIDU_Sock_Wait() > >> MPIDI_CH3I_Progress_handle_sock_event(697)..: > >> MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to > >> connnect to remote process > >> MPIDU_Socki_handle_connect(808).............: connection failure > >> (set=0,sock=1,errno=113:No route to host) > >> [proxy:0:0@tab] got pmi command (from 0): abort > >> exitcode=69331543 > >> [proxy:0:0@tab] we don't understand this command abort; forwarding > upstream > >> [mpiexec@tab] [pgid: 0] got PMI command: cmd=abort exitcode=69331543 > >> > >> > >> > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_Comm_disconnect and pending communications
by Jorge Bellón 26 Jul '16

26 Jul '16

Hi, I am currently working with dynamic process creation with MPI using MPI_Comm_spawn_multiple. However, I'm facing issues during the application cleanup: MPI_Comm_disconnect function is hung and does not ever return (it is being called by all the processes involved in the intercommunicator). I know this issue could happen when there are unfinished communications (this has already happened to me in the past), in a similar way MPI_Finalize waits until all the communications in all the communicators are completed. This time, though, I'm not able to track which communications haven't finished yet. I was able to trace all the calls to the library using gdb and breakpoints, but everything seems OK to me. Is there any way to log the start and arrival of the messages (maybe using configure "--enable-g" option or setting one of the environment variables)? Thanks in advance. Regards, Jorge WARNING / LEGAL TEXT: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. http://www.bsc.es/disclaimer _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error creating 272 processes on a multicore, single CPU
by Gajbe, Manisha 26 Jul '16

26 Jul '16

Hi Kenneth, The configure script is default except the "prefix". I used Intel compilers version 16.0.2. $ ./configure --prefix=/usr/local/my-mpich-3.2/64bit Hi Halim, I removed all the writes to stdout. No change in the output. Also, the code runs with IntelMPI 5.1.3 without any issues with lots of writes to stdout. Do you have any suggestion on setting up the limits of pipe on what should be the reasonable size. Below is output from my ulimit [mmgajbe@fm05wcon0025 Test]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 62912 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 2048 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Hi Husen, I tried on multiple systems with different Oss such as RHEL 7.0, OpenSuse, Ubuntu 12.04 etc. However, the error is observed only with MPICH and not with IntelMPI. ~ Manisha -----Original Message----- From: discuss-request(a)mpich.org [mailto:[email protected]] Sent: Tuesday, July 26, 2016 9:00 AM To: discuss(a)mpich.org Subject: discuss Digest, Vol 45, Issue 7 Send discuss mailing list submissions to discuss(a)mpich.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.mpich.org/mailman/listinfo/discuss or, via email, send a message with subject or body 'help' to discuss-request(a)mpich.org You can reach the person managing the list at discuss-owner(a)mpich.org When replying, please edit your Subject line so it is more specific than "Re: Contents of discuss digest..." Today's Topics: 1. Re: Error creating 272 processes on a multicore, single CPU (Husen R) 2. Segfault with MPICH 3.2+Clang but not GCC (Andreas Noack) 3. Re: Segfault with MPICH 3.2+Clang but not GCC (Jeff Hammond) 4. Re: Segfault with MPICH 3.2+Clang but not GCC (Rob Latham) ---------------------------------------------------------------------- Message: 1 Date: Tue, 26 Jul 2016 10:58:01 +0700 From: Husen R <hus3nr(a)gmail.com> To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Error creating 272 processes on a multicore, single CPU Message-ID: <CACPfdUsN7N5TwU++tsoPiUGA1kiS3Cm-VGBaXotDfkKcquhiag(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" it seems the number of processes that can be created in one machine limited by Operating System. On Sat, Jul 23, 2016 at 11:19 AM, Gajbe, Manisha <manisha.gajbe(a)intel.com> wrote: > Hi, > > > > I have installed mpich-3.2 on a multicore platform. When I spawn 272 > processes , I get the error message mentioned below. I am able to > create upto 271 processes successfully. > > > > /usr/local/my-mpich-3.2/64bit/bin/mpirun -n 272 ./hello_c > > > > [cli_0]: write_line: message string doesn't end in newline: :cmd=put > kvsname=kvs_8846_0 key=r2h1 > value=r0#0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271$: > > > > > > > > *Manisha Gajbe* > > *MVE PQV Content* > > SCE ? System Content Engineering > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Post Graduate Student Faculty of Computer Science University of Indonesia Depok

1 0

Re: [mpich-discuss] Segfault with MPICH 3.2+Clang but not GCC
by Jeff Hammond 26 Jul '16

26 Jul '16

Indeed, Rob is right. I only tested np<=2. Didn't see failure until nproc=16 because of how malloc works but in any case, it is trivial to fix this by allocating C to be nproc elements. Jeff #include <mpi.h> #include <stdio.h> #include <stdlib.h> int main(int argc, char *argv[]) { MPI_Init(&argc, &argv); MPI_Comm comm = MPI_COMM_WORLD; uint64_t *A, *C; int rnk, siz; MPI_Comm_rank(comm, &rnk); MPI_Comm_size(comm, &siz); A = calloc(1, sizeof(uint64_t)); C = calloc(siz, sizeof(uint64_t)); A[0] = rnk + 1; MPI_Allgather(A, 1, MPI_UINT64_T, C, 1, MPI_UINT64_T, comm); free(C); free(A); MPI_Finalize(); return 0; } On Tue, Jul 26, 2016 at 9:00 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 07/26/2016 10:17 AM, Andreas Noack wrote: > >> On my El Capitan macbook I get a segfault when running the program below >> with more than a single process but only when MPICH has been compiled >> with Clang. >> >> I don't get that good debug info but here is some of what I got >> > > > valgrind is pretty good at sussing out these sorts of things: > > ==18132== Unaddressable byte(s) found during client check request > ==18132== at 0x504D1D7: MPIR_Localcopy (helper_fns.c:84) > ==18132== by 0x4EC8EA1: MPIR_Allgather_intra (allgather.c:169) > ==18132== by 0x4ECA5EC: MPIR_Allgather (allgather.c:791) > ==18132== by 0x4ECA7A4: MPIR_Allgather_impl (allgather.c:832) > ==18132== by 0x4EC8B5C: MPID_Allgather (mpid_coll.h:61) > ==18132== by 0x4ECB9F7: PMPI_Allgather (allgather.c:978) > ==18132== by 0x4008F5: main (noack_segv.c:18) > ==18132== Address 0x6f2f138 is 8 bytes after a block of size 16 alloc'd > ==18132== at 0x4C2FB55: calloc (in > /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) > ==18132== by 0x4008B0: main (noack_segv.c:15) > ==18132== > ==18132== Invalid write of size 8 > ==18132== at 0x4C326CB: memcpy@@GLIBC_2.14 (in > /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) > ==18132== by 0x504D31B: MPIR_Localcopy (helper_fns.c:84) > ==18132== by 0x4EC8EA1: MPIR_Allgather_intra (allgather.c:169) > ==18132== by 0x4ECA5EC: MPIR_Allgather (allgather.c:791) > ==18132== by 0x4ECA7A4: MPIR_Allgather_impl (allgather.c:832) > ==18132== by 0x4EC8B5C: MPID_Allgather (mpid_coll.h:61) > ==18132== by 0x4ECB9F7: PMPI_Allgather (allgather.c:978) > ==18132== by 0x4008F5: main (noack_segv.c:18) > ==18132== Address 0x6f2f138 is 8 bytes after a block of size 16 alloc'd > ==18132== at 0x4C2FB55: calloc (in > /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) > ==18132== by 0x4008B0: main (noack_segv.c:15) > > > >> MPI_Comm_rank(comm, &rnk); >> A = calloc(1, sizeof(uint64_t)); >> C = calloc(2, sizeof(uint64_t)); >> A[0] = rnk + 1; >> >> MPI_Allgather(A, 1, MPI_UINT64_T, C, 1, MPI_UINT64_T, comm); >> > > Your 'buf count tuple' is ok for A: every process sends one uint64 > > your 'buf count tuple' is too small for C if there are any more than 2 > proceses . > > When you say "more than one"... do you mean 2? > > ==rob > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Segfault with MPICH 3.2+Clang but not GCC
by Rob Latham 26 Jul '16

26 Jul '16

On 07/26/2016 10:17 AM, Andreas Noack wrote: > On my El Capitan macbook I get a segfault when running the program below > with more than a single process but only when MPICH has been compiled > with Clang. > > I don't get that good debug info but here is some of what I got valgrind is pretty good at sussing out these sorts of things: ==18132== Unaddressable byte(s) found during client check request ==18132== at 0x504D1D7: MPIR_Localcopy (helper_fns.c:84) ==18132== by 0x4EC8EA1: MPIR_Allgather_intra (allgather.c:169) ==18132== by 0x4ECA5EC: MPIR_Allgather (allgather.c:791) ==18132== by 0x4ECA7A4: MPIR_Allgather_impl (allgather.c:832) ==18132== by 0x4EC8B5C: MPID_Allgather (mpid_coll.h:61) ==18132== by 0x4ECB9F7: PMPI_Allgather (allgather.c:978) ==18132== by 0x4008F5: main (noack_segv.c:18) ==18132== Address 0x6f2f138 is 8 bytes after a block of size 16 alloc'd ==18132== at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==18132== by 0x4008B0: main (noack_segv.c:15) ==18132== ==18132== Invalid write of size 8 ==18132== at 0x4C326CB: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==18132== by 0x504D31B: MPIR_Localcopy (helper_fns.c:84) ==18132== by 0x4EC8EA1: MPIR_Allgather_intra (allgather.c:169) ==18132== by 0x4ECA5EC: MPIR_Allgather (allgather.c:791) ==18132== by 0x4ECA7A4: MPIR_Allgather_impl (allgather.c:832) ==18132== by 0x4EC8B5C: MPID_Allgather (mpid_coll.h:61) ==18132== by 0x4ECB9F7: PMPI_Allgather (allgather.c:978) ==18132== by 0x4008F5: main (noack_segv.c:18) ==18132== Address 0x6f2f138 is 8 bytes after a block of size 16 alloc'd ==18132== at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==18132== by 0x4008B0: main (noack_segv.c:15) > > MPI_Comm_rank(comm, &rnk); > A = calloc(1, sizeof(uint64_t)); > C = calloc(2, sizeof(uint64_t)); > A[0] = rnk + 1; > > MPI_Allgather(A, 1, MPI_UINT64_T, C, 1, MPI_UINT64_T, comm); Your 'buf count tuple' is ok for A: every process sends one uint64 your 'buf count tuple' is too small for C if there are any more than 2 proceses . When you say "more than one"... do you mean 2? ==rob _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Segfault with MPICH 3.2+Clang but not GCC
by Jeff Hammond 26 Jul '16

26 Jul '16

I cannot reproduce this. I am using Darwin 15.5.0 instead of 15.6.0, but the compiler is identical. I am using MPICH Git master from June 29. At this point, it is unclear to me if the bug is in MPICH or Clang. Jeff vsanthan-mobl1:BUGS jrhammon$ /opt/mpich/dev/clang/default/bin/mpichversion MPICH Version: 3.2 MPICH Release date: unreleased development copy MPICH Device: ch3:nemesis MPICH configure: CC=clang CXX=clang++ FC=false F77=false --enable-cxx --disable-fortran --with-pm=hydra --prefix=/opt/mpich/dev/clang/default --enable-cxx --enable-wrapper-rpath --disable-static --enable-shared MPICH CC: clang -O2 MPICH CXX: clang++ -O2 MPICH F77: false MPICH FC: false vsanthan-mobl1:BUGS jrhammon$ /opt/mpich/dev/clang/default/bin/mpicc -v mpicc for MPICH version 3.2 Apple LLVM version 7.3.0 (clang-703.0.31) Target: x86_64-apple-darwin15.5.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin clang: warning: argument unused during compilation: '-I /opt/mpich/dev/clang/default/include' On Tue, Jul 26, 2016 at 8:17 AM, Andreas Noack <andreasnoackjensen(a)gmail.com > wrote: > On my El Capitan macbook I get a segfault when running the program below > with more than a single process but only when MPICH has been compiled with > Clang. > > I don't get that good debug info but here is some of what I got > > (lldb) c > Process 61129 resuming > Process 61129 stopped > * thread #1: tid = 0x32c438, 0x00000003119d0432 > libpmpi.12.dylib`MPID_Request_create + 244, queue = > 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT) > frame #0: 0x00000003119d0432 libpmpi.12.dylib`MPID_Request_create + 244 > libpmpi.12.dylib`MPID_Request_create: > -> 0x3119d0432 <+244>: movaps %xmm0, 0x230(%rax) > 0x3119d0439 <+251>: movq $0x0, 0x240(%rax) > 0x3119d0444 <+262>: movl %ecx, 0x210(%rax) > 0x3119d044a <+268>: popq %rbp > > My version of Clang is > > Apple LLVM version 7.3.0 (clang-703.0.31) > Target: x86_64-apple-darwin15.6.0 > Thread model: posix > InstalledDir: /Library/Developer/CommandLineTools/usr/bin > > and the bug has been confirmed by my colleague who is running Linux and > compiling with Clang 3.8. The program runs fine with OpenMPI+Clang. > > #include <mpi.h> > #include <stdio.h> > #include <stdlib.h> > > int main(int argc, char *argv[]) > { > MPI_Init(&argc, &argv); > > MPI_Comm comm = MPI_COMM_WORLD; > uint64_t *A, *C; > int rnk; > > MPI_Comm_rank(comm, &rnk); > A = calloc(1, sizeof(uint64_t)); > C = calloc(2, sizeof(uint64_t)); > A[0] = rnk + 1; > > MPI_Allgather(A, 1, MPI_UINT64_T, C, 1, MPI_UINT64_T, comm); > > MPI_Finalize(); > return 0; > } > > > Best regards > > Andreas Noack > Postdoctoral Associate > Computer Science and Artificial Intelligence Laboratory > Massachusetts Institute of Technology > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Segfault with MPICH 3.2+Clang but not GCC
by Andreas Noack 26 Jul '16

26 Jul '16

On my El Capitan macbook I get a segfault when running the program below with more than a single process but only when MPICH has been compiled with Clang. I don't get that good debug info but here is some of what I got (lldb) c Process 61129 resuming Process 61129 stopped * thread #1: tid = 0x32c438, 0x00000003119d0432 libpmpi.12.dylib`MPID_Request_create + 244, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT) frame #0: 0x00000003119d0432 libpmpi.12.dylib`MPID_Request_create + 244 libpmpi.12.dylib`MPID_Request_create: -> 0x3119d0432 <+244>: movaps %xmm0, 0x230(%rax) 0x3119d0439 <+251>: movq $0x0, 0x240(%rax) 0x3119d0444 <+262>: movl %ecx, 0x210(%rax) 0x3119d044a <+268>: popq %rbp My version of Clang is Apple LLVM version 7.3.0 (clang-703.0.31) Target: x86_64-apple-darwin15.6.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin and the bug has been confirmed by my colleague who is running Linux and compiling with Clang 3.8. The program runs fine with OpenMPI+Clang. #include <mpi.h> #include <stdio.h> #include <stdlib.h> int main(int argc, char *argv[]) { MPI_Init(&argc, &argv); MPI_Comm comm = MPI_COMM_WORLD; uint64_t *A, *C; int rnk; MPI_Comm_rank(comm, &rnk); A = calloc(1, sizeof(uint64_t)); C = calloc(2, sizeof(uint64_t)); A[0] = rnk + 1; MPI_Allgather(A, 1, MPI_UINT64_T, C, 1, MPI_UINT64_T, comm); MPI_Finalize(); return 0; } Best regards Andreas Noack Postdoctoral Associate Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error creating 272 processes on a multicore, single CPU
by Husen R 25 Jul '16

25 Jul '16

it seems the number of processes that can be created in one machine limited by Operating System. On Sat, Jul 23, 2016 at 11:19 AM, Gajbe, Manisha <manisha.gajbe(a)intel.com> wrote: > Hi, > > > > I have installed mpich-3.2 on a multicore platform. When I spawn 272 > processes , I get the error message mentioned below. I am able to create > upto 271 processes successfully. > > > > /usr/local/my-mpich-3.2/64bit/bin/mpirun -n 272 ./hello_c > > > > [cli_0]: write_line: message string doesn't end in newline: :cmd=put > kvsname=kvs_8846_0 key=r2h1 > value=r0#0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271$: > > > > > > > > *Manisha Gajbe* > > *MVE PQV Content* > > SCE – System Content Engineering > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Post Graduate Student Faculty of Computer Science University of Indonesia Depok _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error while using MPI_Send
by Balaji, Pavan 25 Jul '16

25 Jul '16

This sometimes happens if the node cannot resolve its hostname itself. Can you try adding your hostname and IP to /etc/hosts? - Pavan > On Jul 25, 2016, at 1:36 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > > Hi, > > Normally this type of error would be the result of a firewall blocking communication, but your run is on a single node so that shouldn't be the case. I wonder, since it did work at one point, if there is some bad state on your system that might be cleared with a reboot? > > It also looks like you are running on a Android system, which we do not have much experience with, and no good way to test ourselves so our ways to help may be limited. > > Ken > > On 07/24/2016 11:50 AM, Doha Ehab wrote: >> Hello >> I am using a cross compiled version of MPICH3, I was trying simple >> containing MPI_Send and MPI_Recv, it was working but suddenly I keep >> receiving this error messsage: can anyone point out what is wrong and >> how to fix it. >> >> $ mpiexec -v -n 4 /data/parallelCode >> host: tab >> >> ================================================================================================== >> mpiexec options: >> ---------------- >> Base path: /system/xbin/ >> Launcher: (null) >> Debug level: 1 >> Enable X: -1 >> >> Global environment: >> ------------------- >> _=/system/xbin/mpiexec >> PATH=/sbin:/vendor/bin:/system/sbin:/system/bin:/system/xbin >> LOOP_MOUNTPOINT=/mnt/obb >> ANDROID_ROOT=/system >> SHELL=/system/bin/sh >> ANDROID_DATA=/data >> ANDROID_ASSETS=/system/app >> TERM=vt100 >> ANDROID_PROPERTY_WORKSPACE=8,0 >> ANDROID_BOOTLOGO=1 >> HOSTNAME=hwt1701 >> LD_LIBRARY_PATH=/vendor/lib:/system/lib >> >> BOOTCLASSPATH=/system/framework/core.jar:/system/framework/conscrypt.jar:/system/framework/okhttp.jar:/system/framework/core-junit.jar:/system/framework/bouncycastle.jar:/system/framework/ext.jar:/system/framework/framework.jar:/system/framework/framework2.jar:/system/framework/hwframework.jar:/system/framework/hwcustframework.jar:/system/framework/telephony-common.jar:/system/framework/voip-common.jar:/system/framework/mms-common.jar:/system/framework/android.policy.jar:/system/framework/services.jar:/system/framework/apache-xml.jar:/system/framework/webviewchromium.jar:/system/framework/hwEmui.jar:/system/framework/hwServices.jar:/system/framework/hwAndroid.policy.jar:/system/framework/hwTelephony-common.jar:/system/framework/hwpadext.jar >> EMULATED_STORAGE_SOURCE=/mnt/shell/emulated >> ANDROID_SOCKET_adbd=10 >> EMULATED_STORAGE_TARGET=/storage/emulated >> ANDROID_STORAGE=/storage >> MKSH=/system/bin/sh >> EXTERNAL_STORAGE=/storage/emulated/legacy >> USBHOST_STORAGE=/storage/usbdisk >> RANDOM=11338 >> ASEC_MOUNTPOINT=/mnt/asec >> SECONDARY_STORAGE=/storage/sdcard1 >> USER=shell >> LEGACY_STORAGE=/storage/emulated/legacy >> HOME=/data >> >> Hydra internal environment: >> --------------------------- >> GFORTRAN_UNBUFFERED_PRECONNECTED=y >> >> >> Proxy information: >> ********************* >> [1] proxy: tab (1 cores) >> Exec list: /data/mmp100 (4 processes); >> >> >> ================================================================================================== >> >> [mpiexec@tab] Timeout set to -1 (-1 means infinite) >> [mpiexec@tab] Got a control port string of tab:48661 >> >> Proxy launch args: /system/xbin/hydra_pmi_proxy --control-port tab:48661 >> --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 >> --usize -2 --proxy-id >> >> Arguments being passed to proxy 0: >> --version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME >> --hostname tab --global-core-map 0,1,1 --pmi-id-map 0,0 >> --global-process-count 4 --auto-cleanup 1 --pmi-kvsname kvs_10003_0 >> --pmi-process-mapping (vector,(0,1,1)) --ckpoint-num -1 >> --global-inherited-env 26 '_=/system/xbin/mpiexec' >> 'PATH=/sbin:/vendor/bin:/system/sbin:/system/bin:/system/xbin' >> 'LOOP_MOUNTPOINT=/mnt/obb' 'ANDROID_ROOT=/system' 'SHELL=/system/bin/sh' >> 'ANDROID_DATA=/data' 'ANDROID_ASSETS=/system/app' 'TERM=vt100' >> 'ANDROID_PROPERTY_WORKSPACE=8,0' 'ANDROID_BOOTLOGO=1' 'HOSTNAME=hwt1701' >> 'LD_LIBRARY_PATH=/vendor/lib:/system/lib' >> 'BOOTCLASSPATH=/system/framework/core.jar:/system/framework/conscrypt.jar:/system/framework/okhttp.jar:/system/framework/core-junit.jar:/system/framework/bouncycastle.jar:/system/framework/ext.jar:/system/framework/framework.jar:/system/framework/framework2.jar:/system/framework/hwframework.jar:/system/framework/hwcustframework.jar:/system/framework/telephony-common.jar:/system/framework/voip-common.jar:/system/framework/mms-common.jar:/system/framework/android.policy.jar:/system/framework/services.jar:/system/framework/apache-xml.jar:/system/framework/webviewchromium.jar:/system/framework/hwEmui.jar:/system/framework/hwServices.jar:/system/framework/hwAndroid.policy.jar:/system/framework/hwTelephony-common.jar:/system/framework/hwpadext.jar' >> 'EMULATED_STORAGE_SOURCE=/mnt/shell/emulated' 'ANDROID_SOCKET_adbd=10' >> 'EMULATED_STORAGE_TARGET=/storage/emulated' 'ANDROID_STORAGE=/storage' >> 'MKSH=/system/bin/sh' 'EXTERNAL_STORAGE=/storage/emulated/legacy' >> 'USBHOST_STORAGE=/storage/usbdisk' 'RANDOM=11338' >> 'ASEC_MOUNTPOINT=/mnt/asec' 'SECONDARY_STORAGE=/storage/sdcard1' >> 'USER=shell' 'LEGACY_STORAGE=/storage/emulated/legacy' 'HOME=/data' >> --global-user-env 0 --global-system-env 1 >> 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec >> --exec-appnum 0 --exec-proc-count 4 --exec-local-env 0 --exec-wdir / >> --exec-args 1 /data/mmp100 >> >> [mpiexec@tab] Launch arguments: /system/xbin/hydra_pmi_proxy >> --control-port tab:48661 --debug --rmk user --launcher ssh --demux poll >> --pgid 0 --retries 10 --usize -2 --proxy-id 0 >> [proxy:0:0@tab] got pmi command (from 0): init >> pmi_version=1 pmi_subversion=1 >> [proxy:0:0@tab] PMI response: cmd=response_to_init pmi_version=1 >> pmi_subversion=1 rc=0 >> [proxy:0:0@tab] got pmi command (from 0): get_maxes >> >> [proxy:0:0@tab] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 >> vallen_max=1024 >> [proxy:0:0@tab] got pmi command (from 6): init >> pmi_version=1 pmi_subversion=1 >> [proxy:0:0@tab] PMI response: cmd=response_to_init pmi_version=1 >> pmi_subversion=1 rc=0 >> [proxy:0:0@tab] got pmi command (from 6): get_maxes >> >> [proxy:0:0@tab] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 >> vallen_max=1024 >> [proxy:0:0@tab] got pmi command (from 9): init >> pmi_version=1 pmi_subversion=1 >> [proxy:0:0@tab] PMI response: cmd=response_to_init pmi_version=1 >> pmi_subversion=1 rc=0 >> [proxy:0:0@tab] got pmi command (from 15): init >> pmi_version=1 pmi_subversion=1 >> [proxy:0:0@tab] PMI response: cmd=response_to_init pmi_version=1 >> pmi_subversion=1 rc=0 >> [proxy:0:0@tab] got pmi command (from 0): get_appnum >> >> [proxy:0:0@tab] PMI response: cmd=appnum appnum=0 >> [proxy:0:0@tab] got pmi command (from 9): get_maxes >> >> [proxy:0:0@tab] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 >> vallen_max=1024 >> [proxy:0:0@tab] got pmi command (from 0): get_my_kvsname >> >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 >> [proxy:0:0@tab] got pmi command (from 15): get_maxes >> >> [proxy:0:0@tab] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 >> vallen_max=1024 >> [proxy:0:0@tab] got pmi command (from 0): get_my_kvsname >> >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 >> [proxy:0:0@tab] got pmi command (from 9): get_appnum >> >> [proxy:0:0@tab] PMI response: cmd=appnum appnum=0 >> [proxy:0:0@tab] got pmi command (from 0): get >> kvsname=kvs_10003_0 key=PMI_process_mapping >> [proxy:0:0@tab] PMI response: cmd=get_result rc=0 msg=success >> value=(vector,(0,1,1)) >> [proxy:0:0@tab] got pmi command (from 15): get_appnum >> >> [proxy:0:0@tab] PMI response: cmd=appnum appnum=0 >> [proxy:0:0@tab] got pmi command (from 6): get_appnum >> >> [proxy:0:0@tab] PMI response: cmd=appnum appnum=0 >> [proxy:0:0@tab] got pmi command (from 9): get_my_kvsname >> >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 >> [proxy:0:0@tab] got pmi command (from 15): get_my_kvsname >> >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 >> [proxy:0:0@tab] got pmi command (from 6): get_my_kvsname >> >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 >> [proxy:0:0@tab] got pmi command (from 6): get_my_kvsname >> >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 >> [proxy:0:0@tab] got pmi command (from 9): get_my_kvsname >> >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 >> [proxy:0:0@tab] got pmi command (from 0): put >> kvsname=kvs_10003_0 key=P0-businesscard >> value=port#49751$description#tab$ifname#192.168.1.4$ >> [proxy:0:0@tab] cached command: >> P0-businesscard=port#49751$description#tab$ifname#192.168.1.4$ >> [proxy:0:0@tab] PMI response: cmd=put_result rc=0 msg=success >> [proxy:0:0@tab] got pmi command (from 9): get >> kvsname=kvs_10003_0 key=PMI_process_mapping >> [proxy:0:0@tab] PMI response: cmd=get_result rc=0 msg=success >> value=(vector,(0,1,1)) >> [proxy:0:0@tab] got pmi command (from 0): barrier_in >> >> [proxy:0:0@tab] got pmi command (from 6): get >> kvsname=kvs_10003_0 key=PMI_process_mapping >> [proxy:0:0@tab] PMI response: cmd=get_result rc=0 msg=success >> value=(vector,(0,1,1)) >> [proxy:0:0@tab] got pmi command (from 15): get_my_kvsname >> >> [proxy:0:0@tab] PMI response: cmd=my_kvsname kvsname=kvs_10003_0 >> [proxy:0:0@tab] got pmi command (from 9): put >> kvsname=kvs_10003_0 key=P2-businesscard >> value=port#60729$description#tab$ifname#192.168.1.4$ >> [proxy:0:0@tab] cached command: >> P2-businesscard=port#60729$description#tab$ifname#192.168.1.4$ >> [proxy:0:0@tab] PMI response: cmd=put_result rc=0 msg=success >> [proxy:0:0@tab] got pmi command (from 15): get >> kvsname=kvs_10003_0 key=PMI_process_mapping >> [proxy:0:0@tab] PMI response: cmd=get_result rc=0 msg=success >> value=(vector,(0,1,1)) >> [proxy:0:0@tab] got pmi command (from 9): barrier_in >> >> [proxy:0:0@tab] got pmi command (from 6): put >> kvsname=kvs_10003_0 key=P1-businesscard >> value=port#44344$description#tab$ifname#192.168.1.4$ >> [proxy:0:0@tab] cached command: >> P1-businesscard=port#44344$description#tab$ifname#192.168.1.4$ >> [proxy:0:0@tab] PMI response: cmd=put_result rc=0 msg=success >> [proxy:0:0@tab] got pmi command (from 15): put >> kvsname=kvs_10003_0 key=P3-businesscard >> value=port#51326$description#tab$ifname#192.168.1.4$ >> [proxy:0:0@tab] cached command: >> P3-businesscard=port#51326$description#tab$ifname#192.168.1.4$ >> [proxy:0:0@tab] PMI response: cmd=put_result rc=0 msg=success >> [proxy:0:0@tab] got pmi command (from 6): barrier_in >> >> [proxy:0:0@tab] got pmi command (from 15): barrier_in >> >> [proxy:0:0@tab] flushing 4 put command(s) out >> [mpiexec@tab] [pgid: 0] got PMI command: cmd=put >> P0-businesscard=port#49751$description#tab$ifname#192.168.1.4$ >> P2-businesscard=port#60729$description#tab$ifname#192.168.1.4$ >> P1-businesscard=port#44344$description#tab$ifname#192.168.1.4$ >> P3-businesscard=port#51326$description#tab$ifname#192.168.1.4$ >> [proxy:0:0@tab] forwarding command (cmd=put >> P0-businesscard=port#49751$description#tab$ifname#192.168.1.4$ >> P2-businesscard=port#60729$description#tab$ifname#192.168.1.4$ >> P1-businesscard=port#44344$description#tab$ifname#192.168.1.4$ >> P3-businesscard=port#51326$description#tab$ifname#192.168.1.4$) upstream >> [proxy:0:0@tab] forwarding command (cmd=barrier_in) upstream >> [mpiexec@tab] [pgid: 0] got PMI command: cmd=barrier_in >> [mpiexec@tab] PMI response to fd 6 pid 15: cmd=keyval_cache >> P0-businesscard=port#49751$description#tab$ifname#192.168.1.4$ >> P2-businesscard=port#60729$description#tab$ifname#192.168.1.4$ >> P1-businesscard=port#44344$description#tab$ifname#192.168.1.4$ >> P3-businesscard=port#51326$description#tab$ifname#192.168.1.4$ >> [mpiexec@tab] PMI response to fd 6 pid 15: cmd=barrier_out >> [proxy:0:0@tab] PMI response: cmd=barrier_out >> [proxy:0:0@tab] PMI response: cmd=barrier_out >> [proxy:0:0@tab] PMI response: cmd=barrier_out >> [proxy:0:0@tab] PMI response: cmd=barrier_out >> [proxy:0:0@tab] got pmi command (from 0): get >> kvsname=kvs_10003_0 key=P1-businesscard >> [proxy:0:0@tab] PMI response: cmd=get_result rc=0 msg=success >> value=port#44344$description#tab$ifname#192.168.1.4$ >> Fatal error in MPI_Send: Unknown error class, error stack: >> MPI_Send(174)...............................: MPI_Send(buf=0x15c56c, >> count=1, MPI_INT, dest=1, tag=1, MPI_COMM_WORLD) failed >> MPIDI_CH3i_Progress_wait(242)...............: an error occurred while >> handling an event returned by MPIDU_Sock_Wait() >> MPIDI_CH3I_Progress_handle_sock_event(697)..: >> MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to >> connnect to remote process >> MPIDU_Socki_handle_connect(808).............: connection failure >> (set=0,sock=1,errno=113:No route to host) >> [proxy:0:0@tab] got pmi command (from 0): abort >> exitcode=69331543 >> [proxy:0:0@tab] we don't understand this command abort; forwarding upstream >> [mpiexec@tab] [pgid: 0] got PMI command: cmd=abort exitcode=69331543 >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0