November 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] discuss Digest, Vol 48, Issue 6
by Mahdi, Sam 01 Nov '16

01 Nov '16

Hello everyone, The host file content is just the IP of the other computer. So its just: 130.166.115.232 When I typed in the mpirun --host my comp,therecomp /bin/hostname this was the output crowlab: [~]> mpirun --host 130.166.115.232,130.166.115.232 /bin/hostname [proxy:0:[email protected]] HYDU_sock_connect (./utils/sock/sock.c:174): unable to connect from "localhost.localdomain" to "localhost.localdomain" (Connection refused) [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:189): unable to connect to server localhost.localdomain at port 41231 (check for firewalls!) Again same problem as before, its attempting to connect from "localhost.localdomain" (my computer) to "localhost.localdomain" (again my computer) instead of using the IP addresses that I gave it. Sincerely, Sam On Mon, Oct 31, 2016 at 9:47 PM, <discuss-request(a)mpich.org> wrote: > Send discuss mailing list submissions to > discuss(a)mpich.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mpich.org/mailman/listinfo/discuss > or, via email, send a message with subject or body 'help' to > discuss-request(a)mpich.org > > You can reach the person managing the list at > discuss-owner(a)mpich.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of discuss digest..." > > > Today's Topics: > > 1. Re: Running a program on multiple computers (Kenneth Raffenetti) > 2. Re: Using external load-balancer with mpich (Halim Amer) > 3. Re: Using external load-balancer with mpich > (lostfreeman(a)gmail.com) > 4. Re: Is there a way to set timeout to mpi process launch? > (Halim Amer) > 5. Re: Using external load-balancer with mpich (Halim Amer) > 6. Re: Is there a way to set timeout to mpi process launch? > (Pranav Ladkat) > 7. ADIOI_Set_lock error (Luke Van Roekel) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 27 Oct 2016 22:17:59 -0500 > From: Kenneth Raffenetti <raffenet(a)mcs.anl.gov> > To: <discuss(a)mpich.org> > Subject: Re: [mpich-discuss] Running a program on multiple computers > Message-ID: <3a59ae1a-8577-4b86-3e99-b27d97544ac6(a)mcs.anl.gov> > Content-Type: text/plain; charset="utf-8"; format=flowed > > Hi, > > Apologies for the delay in response. My comments inline below. > > On 10/15/2016 07:40 PM, Mahdi, Sam wrote: > > Hello everyone, > > > > I am attempting to run a single program on 32 cores split across 4 > > computers (So each computer has 8 cores). I am attempting to use mpich > > for this. I currently am just testing on 2 computers, I have the program > > installed on both, as well as mpich installed on both. I have created a > > register key and can login in using ssh into the other computer without > > a password. I have come across 2 problems. One, when I attempt to > > connect using the mpirun -np 3 --host a (the IP of the computer I am > > attempting to connect to) hostname > > I recieve the error > > unable to connect from "localhost.localdomain" to > "localhost.localdomain" > > > > This is indicating my computers "localhost.localdomain" is attempting to > > connect to another "localhost.localdomain". How can I change this so > > that it connects via my IP to the other computers IP? > > > > Secondly, I attempted to use a host file instead using the hydra process > > wiki. I created a hosts file with just the IP of the computer I am > > attempting to connect to. When I type in the command mpiexec -f hosts -n > > 4 ./applic > > > > I get this error > > [mpiexec(a)localhost.localdomain] HYDU_parse_hostfile > > (./utils/args/args.c:323): unable to open host file: hosts > > > > along with other errors of unable to parse hostfile, match handler etc. > > I assume this is all due to it being unable to read the host file. Is > > there any specific place I should save my hosts file? I have it saved > > directly on my Desktop. I have attempted to indicate the full path where > > it is located, but I still get the same error. > > There is no required location for the hosts file. If you are specifying > full path and there are still issues, it may be a formatting issue. Can > you paste or attach the contents of your hosts file so we can confirm > the format is good? > > > > > For the first problem, I have read that I need to change /etc/hosts > > manually by using the sudo command to manually enter the IP of the > > computer I am attempting to connect to in the /etc/hosts file. I assume > > the computer is attempting to connect to itself (set up the program > > first on its own core, then send it to another, hence attempting to > > start it on localhost.localdomain). > > > > For the second problem, I have attempted to add run the command > > mpirun --host my computer IP, the other computer IP ./program > > This format should be okay for your purposes. What happens if you try: > > mpirun --host my computer IP, the other computer IP /bin/hostname > > If the hostnames of each host are echoed to the command-line, then job > launch is successful and the issues is during connection setup during > MPI_Init. > > Ken > > > ------------------------------ > > Message: 2 > Date: Thu, 27 Oct 2016 23:18:08 -0500 > From: Halim Amer <aamer(a)anl.gov> > To: <discuss(a)mpich.org> > Subject: Re: [mpich-discuss] Using external load-balancer with mpich > Message-ID: <84e884b3-6a11-fdf7-c143-93adc64a4feb(a)anl.gov> > Content-Type: text/plain; charset="windows-1252"; format=flowed > > I don't understand what you are trying to do. Can you give an example? > > Halim > www.mcs.anl.gov/~aamer > > On 10/26/16 5:21 PM, lost wrote: > > Can I use an external load balancer with mpiexec by providing a single > > hostname in hosts file with, optionally, some large number for host > > rank, and putting load balancer listening on that hostname and > > forwarding connections to the actual hosts? > > > > I am trying to achieve autoscaling (load balancer tracks liveness of > > hosts and spins up new ones on demand). > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > ------------------------------ > > Message: 3 > Date: Thu, 27 Oct 2016 21:59:03 -0700 > From: <lostfreeman(a)gmail.com> > To: Halim Amer <aamer(a)anl.gov>, "discuss(a)mpich.org" > <discuss(a)mpich.org> > Subject: Re: [mpich-discuss] Using external load-balancer with mpich > Message-ID: <5812db17.5789620a.c825c.7724(a)mx.google.com> > Content-Type: text/plain; charset="utf-8" > > I have a system, that can start and stop machines depending on fleet?s > current load. Essentially, it must be the one to decide, which machine will > receive a new task, and optionally start acquire a new one from some pool > for it. > > For example, I can have two hosts of six currently executing something > (might be not related to MPICH), and my load balancer machine is aware of > it. Other 4 are in low-power state to conserve costs. So when I call > mpiexec, I want to tell it to send all the tasks to the load balancer, > requesting 2 hosts, so that load balancer then could start two new hosts to > handle that request. If the initial hosts would not be busy with some other > task, load balancer would send incoming jobs to them, and kept all other > four deallocated. > > - Victor > > From: Halim Amer > Sent: Thursday, October 27, 2016 9:18 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Using external load-balancer with mpich > > I don't understand what you are trying to do. Can you give an example? > > Halim > www.mcs.anl.gov/~aamer > > On 10/26/16 5:21 PM, lost wrote: > > Can I use an external load balancer with mpiexec by providing a single > > hostname in hosts file with, optionally, some large number for host > > rank, and putting load balancer listening on that hostname and > > forwarding connections to the actual hosts? > > > > I am trying to achieve autoscaling (load balancer tracks liveness of > > hosts and spins up new ones on demand). > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] ADIOI_Set_lock error
by Rob Latham 01 Nov '16

01 Nov '16

On 11/01/2016 11:10 AM, Wei-keng Liao wrote: > Hi, Luke > > Could you try the attached program and run by adding "lustre:" prefix to > the filename? > I.e. mpiexec -n 2 a.out lustre:/path/to/Lustre/testfile > > This checks whether the Intel MPI calls the Lustre driver correctly. > Intel's MPI does driver selection a little differently. Does prefixing the file name work? http://press3.mcs.anl.gov/romio/2014/06/12/romio-and-intel-mpi/ You request a file system with the I_MPI_EXTRA_FILESYSTEM and I_MPI_EXTRA_FILESYSTEM_LIST environment variables. ==rob > Wei-keng > > > > On Nov 1, 2016, at 10:00 AM, Luke Van Roekel wrote: > >> Thanks Rob, no luck with the mpi_info object and our computing folks are not willing to set the necessary option. Do you know why this is specific to intel-mpi? Openmpi has no issue. Our HPC people thought the MPI_File_write_at_all always requires a file lock, but openmpi seems to be fine. >> >> On Tue, Nov 1, 2016 at 8:42 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: >> >> >> On 10/31/2016 11:47 PM, Luke Van Roekel wrote: >> Hello, >> >> I've been trying to compile and run a very simple mpi test on our >> cluster with intel-mpi and openmpi. The test program is below. When I >> run with openmpi everything is fine. When I run with intel-mpi, I >> receive the following error >> >> This requires fcntl(2) to be implemented. As of 8/25/2011 it is not. >> Generic MPICH Message: File locking failed in ADIOI_Set_lock(fd 6,cmd >> F_SETLKW/7,type F_WRLCK/1,whence 0) with return value FFFFFFFF and errno 26. >> >> >> - If the file system is NFS, you need to use NFS version 3, ensure that >> the lockd daemon is running on all the machines, and mount the directory >> with the 'noac' option (no attribute caching). >> >> - If the file system is LUSTRE, ensure that the directory is mounted >> with the 'flock' option. >> >> ADIOI_Set_lock:: Function not implemented >> >> ADIOI_Set_lock:offset 0, length 4 >> >> >> your site administrator needs to enable fcntl locking with the 'flock' mount option . >> >> You can try disabling data sieving: you would create an MPI_Info object and add the key "romio_ds_write" with the value "disable" >> >> ==rob >> >> >> >> Any thoughts on how to proceed? The size/format of the file read in >> seems to make no difference. >> >> Regards, >> Luke >> >> >> #include <stdio.h> >> >> #include <stdlib.h> >> >> #include <mpi.h> >> >> >> >> int main(int argc, char **argv) { >> >> int buf, err; >> >> MPI_File fh; >> >> MPI_Status status; >> >> >> MPI_Init(&argc, &argv); >> >> if (argc != 2) { >> >> printf("Usage: %s filename\n", argv[0]); >> >> MPI_Finalize(); >> >> return 1; >> >> } >> >> err = MPI_File_open(MPI_COMM_WORLD, argv[1], MPI_MODE_CREATE | >> >> MPI_MODE_RDWR, MPI_INFO_NULL, &fh); >> >> if (err != MPI_SUCCESS) printf("Error: MPI_File_open()\n"); >> >> >> err = MPI_File_write_all(fh, &buf, 1, MPI_INT, &status); >> >> if (err != MPI_SUCCESS) printf("Error: MPI_File_write_all()\n"); >> >> >> MPI_File_close(&fh); >> >> MPI_Finalize(); >> >> return 0; >> >> } >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] ADIOI_Set_lock error
by Luke Van Roekel 01 Nov '16

01 Nov '16

Hello Wei-king, No luck there, still the same error. I'll ping our computer folks again. Luke On Tue, Nov 1, 2016 at 10:10 AM, Wei-keng Liao <wkliao(a)eecs.northwestern.edu > wrote: > Hi, Luke > > Could you try the attached program and run by adding "lustre:" prefix to > the filename? > I.e. mpiexec -n 2 a.out lustre:/path/to/Lustre/testfile > > This checks whether the Intel MPI calls the Lustre driver correctly. > > Wei-keng > > > > > On Nov 1, 2016, at 10:00 AM, Luke Van Roekel wrote: > > > Thanks Rob, no luck with the mpi_info object and our computing folks are > not willing to set the necessary option. Do you know why this is specific > to intel-mpi? Openmpi has no issue. Our HPC people thought the > MPI_File_write_at_all always requires a file lock, but openmpi seems to be > fine. > > > > On Tue, Nov 1, 2016 at 8:42 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > > > > On 10/31/2016 11:47 PM, Luke Van Roekel wrote: > > Hello, > > > > I've been trying to compile and run a very simple mpi test on our > > cluster with intel-mpi and openmpi. The test program is below. When I > > run with openmpi everything is fine. When I run with intel-mpi, I > > receive the following error > > > > This requires fcntl(2) to be implemented. As of 8/25/2011 it is not. > > Generic MPICH Message: File locking failed in ADIOI_Set_lock(fd 6,cmd > > F_SETLKW/7,type F_WRLCK/1,whence 0) with return value FFFFFFFF and errno > 26. > > > > > > - If the file system is NFS, you need to use NFS version 3, ensure that > > the lockd daemon is running on all the machines, and mount the directory > > with the 'noac' option (no attribute caching). > > > > - If the file system is LUSTRE, ensure that the directory is mounted > > with the 'flock' option. > > > > ADIOI_Set_lock:: Function not implemented > > > > ADIOI_Set_lock:offset 0, length 4 > > > > > > your site administrator needs to enable fcntl locking with the 'flock' > mount option . > > > > You can try disabling data sieving: you would create an MPI_Info object > and add the key "romio_ds_write" with the value "disable" > > > > ==rob > > > > > > > > Any thoughts on how to proceed? The size/format of the file read in > > seems to make no difference. > > > > Regards, > > Luke > > > > > > #include <stdio.h> > > > > #include <stdlib.h> > > > > #include <mpi.h> > > > > > > > > int main(int argc, char **argv) { > > > > int buf, err; > > > > MPI_File fh; > > > > MPI_Status status; > > > > > > MPI_Init(&argc, &argv); > > > > if (argc != 2) { > > > > printf("Usage: %s filename\n", argv[0]); > > > > MPI_Finalize(); > > > > return 1; > > > > } > > > > err = MPI_File_open(MPI_COMM_WORLD, argv[1], MPI_MODE_CREATE | > > > > MPI_MODE_RDWR, MPI_INFO_NULL, &fh); > > > > if (err != MPI_SUCCESS) printf("Error: MPI_File_open()\n"); > > > > > > err = MPI_File_write_all(fh, &buf, 1, MPI_INT, &status); > > > > if (err != MPI_SUCCESS) printf("Error: MPI_File_write_all()\n"); > > > > > > MPI_File_close(&fh); > > > > MPI_Finalize(); > > > > return 0; > > > > } > > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] ADIOI_Set_lock error
by Wei-keng Liao 01 Nov '16

01 Nov '16

1 0

Re: [mpich-discuss] ADIOI_Set_lock error
by Rob Latham 01 Nov '16

01 Nov '16

On 11/01/2016 10:00 AM, Luke Van Roekel wrote: > Thanks Rob, no luck with the mpi_info object and our computing folks are > not willing to set the necessary option. Do you know why this is > specific to intel-mpi? Openmpi has no issue. Our HPC people thought > the MPI_File_write_at_all always requires a file lock, but openmpi seems > to be fine. I don't think the collectives (routines ending in _all) will trigger a lock request. ROMIO makes a lock request to guard against false sharing when performing a data sieving write it's called data sieving but it's actually a read-modify-write and we need to protect against two processes doing partial updates of the same region. OpenMPI has a different MPI-IO implementation as of I think 1.10. I don't know what optimizations they implement or how they avoid false sharing. ==rob > > On Tue, Nov 1, 2016 at 8:42 AM, Rob Latham <robl(a)mcs.anl.gov > <mailto:[email protected]>> wrote: > > > > On 10/31/2016 11:47 PM, Luke Van Roekel wrote: > > Hello, > > I've been trying to compile and run a very simple mpi test on our > cluster with intel-mpi and openmpi. The test program is below. > When I > run with openmpi everything is fine. When I run with intel-mpi, I > receive the following error > > This requires fcntl(2) to be implemented. As of 8/25/2011 it is not. > Generic MPICH Message: File locking failed in ADIOI_Set_lock(fd > 6,cmd > F_SETLKW/7,type F_WRLCK/1,whence 0) with return value FFFFFFFF > and errno 26. > > > > - If the file system is NFS, you need to use NFS version 3, > ensure that > the lockd daemon is running on all the machines, and mount the > directory > with the 'noac' option (no attribute caching). > > - If the file system is LUSTRE, ensure that the directory is mounted > with the 'flock' option. > > ADIOI_Set_lock:: Function not implemented > > ADIOI_Set_lock:offset 0, length 4 > > > > your site administrator needs to enable fcntl locking with the > 'flock' mount option . > > You can try disabling data sieving: you would create an MPI_Info > object and add the key "romio_ds_write" with the value "disable" > > ==rob > > > > Any thoughts on how to proceed? The size/format of the file read in > seems to make no difference. > > Regards, > Luke > > > #include <stdio.h> > > #include <stdlib.h> > > #include <mpi.h> > > > > int main(int argc, char **argv) { > > int buf, err; > > MPI_File fh; > > MPI_Status status; > > > MPI_Init(&argc, &argv); > > if (argc != 2) { > > printf("Usage: %s filename\n", argv[0]); > > MPI_Finalize(); > > return 1; > > } > > err = MPI_File_open(MPI_COMM_WORLD, argv[1], MPI_MODE_CREATE | > > MPI_MODE_RDWR, MPI_INFO_NULL, &fh); > > if (err != MPI_SUCCESS) printf("Error: MPI_File_open()\n"); > > > err = MPI_File_write_all(fh, &buf, 1, MPI_INT, &status); > > if (err != MPI_SUCCESS) printf("Error: MPI_File_write_all()\n"); > > > MPI_File_close(&fh); > > MPI_Finalize(); > > return 0; > > } > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] ADIOI_Set_lock error
by Luke Van Roekel 01 Nov '16

01 Nov '16

Thanks Rob, no luck with the mpi_info object and our computing folks are not willing to set the necessary option. Do you know why this is specific to intel-mpi? Openmpi has no issue. Our HPC people thought the MPI_File_write_at_all always requires a file lock, but openmpi seems to be fine. On Tue, Nov 1, 2016 at 8:42 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 10/31/2016 11:47 PM, Luke Van Roekel wrote: > >> Hello, >> >> I've been trying to compile and run a very simple mpi test on our >> cluster with intel-mpi and openmpi. The test program is below. When I >> run with openmpi everything is fine. When I run with intel-mpi, I >> receive the following error >> >> This requires fcntl(2) to be implemented. As of 8/25/2011 it is not. >> Generic MPICH Message: File locking failed in ADIOI_Set_lock(fd 6,cmd >> F_SETLKW/7,type F_WRLCK/1,whence 0) with return value FFFFFFFF and errno >> 26. >> > > >> - If the file system is NFS, you need to use NFS version 3, ensure that >> the lockd daemon is running on all the machines, and mount the directory >> with the 'noac' option (no attribute caching). >> >> - If the file system is LUSTRE, ensure that the directory is mounted >> with the 'flock' option. >> >> ADIOI_Set_lock:: Function not implemented >> >> ADIOI_Set_lock:offset 0, length 4 >> > > > your site administrator needs to enable fcntl locking with the 'flock' > mount option . > > You can try disabling data sieving: you would create an MPI_Info object > and add the key "romio_ds_write" with the value "disable" > > ==rob > > >> >> Any thoughts on how to proceed? The size/format of the file read in >> seems to make no difference. >> >> Regards, >> Luke >> >> >> #include <stdio.h> >> >> #include <stdlib.h> >> >> #include <mpi.h> >> >> >> >> int main(int argc, char **argv) { >> >> int buf, err; >> >> MPI_File fh; >> >> MPI_Status status; >> >> >> MPI_Init(&argc, &argv); >> >> if (argc != 2) { >> >> printf("Usage: %s filename\n", argv[0]); >> >> MPI_Finalize(); >> >> return 1; >> >> } >> >> err = MPI_File_open(MPI_COMM_WORLD, argv[1], MPI_MODE_CREATE | >> >> MPI_MODE_RDWR, MPI_INFO_NULL, &fh); >> >> if (err != MPI_SUCCESS) printf("Error: MPI_File_open()\n"); >> >> >> err = MPI_File_write_all(fh, &buf, 1, MPI_INT, &status); >> >> if (err != MPI_SUCCESS) printf("Error: MPI_File_write_all()\n"); >> >> >> MPI_File_close(&fh); >> >> MPI_Finalize(); >> >> return 0; >> >> } >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Is there a way to set timeout to mpi process launch?
by Maksym Planeta 01 Nov '16

01 Nov '16

Dear Pranav, if it is only about dynamic linking, can't you try to run "mpirun ldd <binary>" first on the binary you want to execute? And if all dependencies are met, you launch the program. On 10/27/2016 01:30 AM, Pranav Ladkat wrote: > Hi, > > When I run mpi program on multiple hosts, if executable fails to start > on any of the host (due to missing library etc. type of reasons), other > hosts just keep waiting for the process to come up. The program just > hangs forever. Is there any way to set a timeout in such cases such that > MPI should abort if not all processes were launched in a given timeout > period? > > Thanks, > Pranav > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Regards, Maksym Planeta _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Is there a way to set timeout to mpi process launch?
by Halim Amer 01 Nov '16

01 Nov '16

There is no way to do that in Hydra. The only way is for the user to set timers in the application and MPI_Abort if MPI_Init is taking too long. It is up to the user to determine what "taking too long" would mean. Halim www.mcs.anl.gov/~aamer On 10/31/16 6:18 PM, Pranav Ladkat wrote: > Specifying MPIEXEC_TIMEOUT would not be possible since the execution times > would vary depending on the different job. Is there any solution available > within mpich? > > Thanks, > Pranav > > > On Thu, Oct 27, 2016 at 10:15 PM, Halim Amer <aamer(a)anl.gov> wrote: > >> You can try setting MPIEXEC_TIMEOUT=<timeout value in seconds> to force >> the job to abort after running for the specified period. This is for the >> whole execution though, not just for the process launching step. >> >> Halim >> www.mcs.anl.gov/~aamer >> >> >> On 10/26/16 6:30 PM, Pranav Ladkat wrote: >> >>> Hi, >>> >>> When I run mpi program on multiple hosts, if executable fails to start >>> on any of the host (due to missing library etc. type of reasons), other >>> hosts just keep waiting for the process to come up. The program just >>> hangs forever. Is there any way to set a timeout in such cases such that >>> MPI should abort if not all processes were launched in a given timeout >>> period? >>> >>> Thanks, >>> Pranav >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] ADIOI_Set_lock error
by Rob Latham 01 Nov '16

01 Nov '16

On 10/31/2016 11:47 PM, Luke Van Roekel wrote: > Hello, > > I've been trying to compile and run a very simple mpi test on our > cluster with intel-mpi and openmpi. The test program is below. When I > run with openmpi everything is fine. When I run with intel-mpi, I > receive the following error > > This requires fcntl(2) to be implemented. As of 8/25/2011 it is not. > Generic MPICH Message: File locking failed in ADIOI_Set_lock(fd 6,cmd > F_SETLKW/7,type F_WRLCK/1,whence 0) with return value FFFFFFFF and errno 26. > > - If the file system is NFS, you need to use NFS version 3, ensure that > the lockd daemon is running on all the machines, and mount the directory > with the 'noac' option (no attribute caching). > > - If the file system is LUSTRE, ensure that the directory is mounted > with the 'flock' option. > > ADIOI_Set_lock:: Function not implemented > > ADIOI_Set_lock:offset 0, length 4 your site administrator needs to enable fcntl locking with the 'flock' mount option . You can try disabling data sieving: you would create an MPI_Info object and add the key "romio_ds_write" with the value "disable" ==rob > > > Any thoughts on how to proceed? The size/format of the file read in > seems to make no difference. > > Regards, > Luke > > > #include <stdio.h> > > #include <stdlib.h> > > #include <mpi.h> > > > > int main(int argc, char **argv) { > > int buf, err; > > MPI_File fh; > > MPI_Status status; > > > MPI_Init(&argc, &argv); > > if (argc != 2) { > > printf("Usage: %s filename\n", argv[0]); > > MPI_Finalize(); > > return 1; > > } > > err = MPI_File_open(MPI_COMM_WORLD, argv[1], MPI_MODE_CREATE | > > MPI_MODE_RDWR, MPI_INFO_NULL, &fh); > > if (err != MPI_SUCCESS) printf("Error: MPI_File_open()\n"); > > > err = MPI_File_write_all(fh, &buf, 1, MPI_INT, &status); > > if (err != MPI_SUCCESS) printf("Error: MPI_File_write_all()\n"); > > > MPI_File_close(&fh); > > MPI_Finalize(); > > return 0; > > } > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

ADIOI_Set_lock error
by Luke Van Roekel 31 Oct '16

31 Oct '16

Hello, I've been trying to compile and run a very simple mpi test on our cluster with intel-mpi and openmpi. The test program is below. When I run with openmpi everything is fine. When I run with intel-mpi, I receive the following error This requires fcntl(2) to be implemented. As of 8/25/2011 it is not. Generic MPICH Message: File locking failed in ADIOI_Set_lock(fd 6,cmd F_SETLKW/7,type F_WRLCK/1,whence 0) with return value FFFFFFFF and errno 26. - If the file system is NFS, you need to use NFS version 3, ensure that the lockd daemon is running on all the machines, and mount the directory with the 'noac' option (no attribute caching). - If the file system is LUSTRE, ensure that the directory is mounted with the 'flock' option. ADIOI_Set_lock:: Function not implemented ADIOI_Set_lock:offset 0, length 4 Any thoughts on how to proceed? The size/format of the file read in seems to make no difference. Regards, Luke #include <stdio.h> #include <stdlib.h> #include <mpi.h> int main(int argc, char **argv) { int buf, err; MPI_File fh; MPI_Status status; MPI_Init(&argc, &argv); if (argc != 2) { printf("Usage: %s filename\n", argv[0]); MPI_Finalize(); return 1; } err = MPI_File_open(MPI_COMM_WORLD, argv[1], MPI_MODE_CREATE | MPI_MODE_RDWR, MPI_INFO_NULL, &fh); if (err != MPI_SUCCESS) printf("Error: MPI_File_open()\n"); err = MPI_File_write_all(fh, &buf, 1, MPI_INT, &status); if (err != MPI_SUCCESS) printf("Error: MPI_File_write_all()\n"); MPI_File_close(&fh); MPI_Finalize(); return 0; } _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0