December 2018 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_Comm_spawn zombies have risen from the dead.
by Zhou, Hui 06 Dec '18

06 Dec '18

Hi Thomas, Sure. While I am finding time looking into this, could you talk a bit more about your application? The dynamic MPI processes is still a new territory despite its presence for many years. One of the reasons is due to lack of exposure for such applications. — Hui Zhou On Dec 6, 2018, at 6:11 AM, Thomas Pak <thomas.pak(a)maths.ox.ac.uk<mailto:[email protected]>> wrote: Hi Hui, That is exactly right, thanks so much for looking into it! Please let me know if you make any progress on this issue, as the application I am developing critically depends on MPI_Comm_spawn. Best wishes, Thomas Pak

1 0

Re: [mpich-discuss] MPI_Comm_spawn zombies have risen from the dead.
by Thomas Pak 06 Dec '18

06 Dec '18

Hi Hui, That is exactly right, thanks so much for looking into it! Please let me know if you make any progress on this issue, as the application I am developing critically depends on MPI_Comm_spawn. Best wishes, Thomas Pak On Dec 6 2018, at 1:57 pm, Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi Thomas, > > To summarize issue: MPICH proxy does not reap processes after its exit? > > I’ll look into the issue. From my peruse of current MPI standards, it does not really specify the behavior after MPI_Finalize. The dynamic process part of MPI support is weak, primary due to lack of application adoption. Regardless, I agree that the proxy creates the process should be responsible to reap its dead children. > > — > Hui Zhou > > > > > > > > > > On Dec 6, 2018, at 3:53 AM, Thomas Pak via discuss <discuss(a)mpich.org (mailto:[email protected])> wrote: > > Dear all, > > Does anyone have any feedback on this issue? The problem still persists in MPICH-3.3 and it seems like it is pointing to a severe flaw in how MPICH handles dynamic process creation. To reiterate, the following short program using MPICH creates zombie processes indefinitely until no more processes can be created. > > """ > > #include <stdio.h> > > #include <mpi.h> > > > > int main(int argc, char *argv[]) { > > // Initialize MPI > > MPI_Init(NULL, NULL); > > > > // Get parent > > MPI_Comm parent; > > MPI_Comm_get_parent(&parent); > > > > // If the process was not spawned > > if (parent == MPI_COMM_NULL) { > > > > puts("I was not spawned!"); > > // Spawn child process in loop > > char *cmd = argv[0]; > > char **cmd_argv = MPI_ARGV_NULL; > > int maxprocs = 1; > > MPI_Info info = MPI_INFO_NULL; > > int root = 0; > > MPI_Comm comm = MPI_COMM_SELF; > > MPI_Comm intercomm; > > int *array_of_errcodes = MPI_ERRCODES_IGNORE; > > > > for (;;) { > > MPI_Comm_spawn(cmd, cmd_argv, maxprocs, info, root, comm, > > &intercomm, array_of_errcodes); > > > > MPI_Comm_disconnect(&intercomm); > > } > > > > // If process was spawned > > } else { > > > > puts("I was spawned!"); > > MPI_Comm_disconnect(&parent); > > } > > > > // Finalize > > MPI_Finalize(); > > > > } > > """ > > > > Thanks in advance for your help. > > Best wishes, > > Thomas Pak > > > > On Oct 16 2018, at 4:57 pm, Thomas Pak <thomas.pak(a)maths.ox.ac.uk (mailto:[email protected])> wrote: > > > > > > Dear all, > > > My MPI application spawns a large number of MPI processes using MPI_Comm_spawn over its total lifetime. Unfortunately, I have experienced that this results in zombie processes of hydra_pmi_proxy accumulating over time. As a result, my MPI application will eventually crash because there are too many zombie processes and no new processes can be created. > > > This issue does not seem to be new; I have found references to similar issues in at least three different places: > > > - [mpich-discuss] MPI_Comm_Spawn causing zombies of hydra_pmi_proxy (https://lists.mpich.org/pipermail/discuss/2013-March/000599.html) (March 2013) > > > - [GitHub issue] hydra_pmi_proxy zombie MPI_comm_spawn #1677 (https://github.com/pmodels/mpich/issues/1677) (October 2016) > > > - [Google Groups] Hanging in spawn in master-slave code and hydra_pmi_proxy zombies (https://groups.google.com/forum/#!msg/mpi4py/A9mN-2UkFf8/2gnUEtArDwAJ) (December 2016) > > > In the first reference, a dirty fix was proposed by Silvan Brändli to simply reap all the zombie processes in a loop at the start of the function HYDU_create_process in src/pm/hydra/utils/launch/launch.c. However, I have found that this comes with its own set of problems and makes my MPI application unstable. > > > > > > In the GitHub issue, the problem was marked as "worksforme". However, this problem persists for me and other machines that I have tried to reproduce it on, so it clearly has not yet been resolved. > > > In the Google Groups thread, the problem was "resolved" by avoiding spawning MPI processes entirely, which is not an option in my case. > > > Pavan Balaji mentions in the 2013 mpi-discuss thread that this issue was a problem "once upon a time". It seems to have risen from the dead again. > > > I have attached a short and self-contained program written in C that reproduces the problem. The program simply spawns child processes using MPI_Comm_spawn in an infinite loop, where each child process exits after writing a message to stdout. I use MPI_Comm_disconnect to disconnect the Intercommunicator, but the same problem occurs when using MPI_Comm_free instead. > > > I have also attached the logfiles generated while building MPICH, as described in the README, except for the file mpich-3.2.1/src/pm/hydra/tools/topo/hwloc/hwloc/config.log, as that was missing. I am using MPICH version 3.2.1, which is the latest stable release at the time of writing. > > > Thanks in advance for your help! > > > Best wishes, > > > Thomas Pak > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org (mailto:[email protected]) > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org (mailto:[email protected]) > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > >

1 0

Re: [mpich-discuss] MPI_Comm_spawn zombies have risen from the dead.
by Zhou, Hui 06 Dec '18

06 Dec '18

Hi Thomas, To summarize issue: MPICH proxy does not reap processes after its exit? I’ll look into the issue. From my peruse of current MPI standards, it does not really specify the behavior after MPI_Finalize. The dynamic process part of MPI support is weak, primary due to lack of application adoption. Regardless, I agree that the proxy creates the process should be responsible to reap its dead children. — Hui Zhou On Dec 6, 2018, at 3:53 AM, Thomas Pak via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Dear all, Does anyone have any feedback on this issue? The problem still persists in MPICH-3.3 and it seems like it is pointing to a severe flaw in how MPICH handles dynamic process creation. To reiterate, the following short program using MPICH creates zombie processes indefinitely until no more processes can be created. """ #include <stdio.h> #include <mpi.h> int main(int argc, char *argv[]) { // Initialize MPI MPI_Init(NULL, NULL); // Get parent MPI_Comm parent; MPI_Comm_get_parent(&parent); // If the process was not spawned if (parent == MPI_COMM_NULL) { puts("I was not spawned!"); // Spawn child process in loop char *cmd = argv[0]; char **cmd_argv = MPI_ARGV_NULL; int maxprocs = 1; MPI_Info info = MPI_INFO_NULL; int root = 0; MPI_Comm comm = MPI_COMM_SELF; MPI_Comm intercomm; int *array_of_errcodes = MPI_ERRCODES_IGNORE; for (;;) { MPI_Comm_spawn(cmd, cmd_argv, maxprocs, info, root, comm, &intercomm, array_of_errcodes); MPI_Comm_disconnect(&intercomm); } // If process was spawned } else { puts("I was spawned!"); MPI_Comm_disconnect(&parent); } // Finalize MPI_Finalize(); } """ Thanks in advance for your help. Best wishes, Thomas Pak On Oct 16 2018, at 4:57 pm, Thomas Pak <thomas.pak(a)maths.ox.ac.uk<mailto:[email protected]>> wrote: Dear all, My MPI application spawns a large number of MPI processes using MPI_Comm_spawn over its total lifetime. Unfortunately, I have experienced that this results in zombie processes of hydra_pmi_proxy accumulating over time. As a result, my MPI application will eventually crash because there are too many zombie processes and no new processes can be created. This issue does not seem to be new; I have found references to similar issues in at least three different places: - [mpich-discuss] MPI_Comm_Spawn causing zombies of hydra_pmi_proxy<https://lists.mpich.org/pipermail/discuss/2013-March/000599.html> (March 2013) - [GitHub issue] hydra_pmi_proxy zombie MPI_comm_spawn #1677<https://github.com/pmodels/mpich/issues/1677> (October 2016) - [Google Groups] Hanging in spawn in master-slave code and hydra_pmi_proxy zombies<https://groups.google.com/forum/#!msg/mpi4py/A9mN-2UkFf8/2gnUEtArDwAJ> (December 2016) In the first reference, a dirty fix was proposed by Silvan Brändli to simply reap all the zombie processes in a loop at the start of the function HYDU_create_process in src/pm/hydra/utils/launch/launch.c. However, I have found that this comes with its own set of problems and makes my MPI application unstable. In the GitHub issue, the problem was marked as "worksforme". However, this problem persists for me and other machines that I have tried to reproduce it on, so it clearly has not yet been resolved. In the Google Groups thread, the problem was "resolved" by avoiding spawning MPI processes entirely, which is not an option in my case. Pavan Balaji mentions in the 2013 mpi-discuss thread that this issue was a problem "once upon a time". It seems to have risen from the dead again. I have attached a short and self-contained program written in C that reproduces the problem. The program simply spawns child processes using MPI_Comm_spawn in an infinite loop, where each child process exits after writing a message to stdout. I use MPI_Comm_disconnect to disconnect the Intercommunicator, but the same problem occurs when using MPI_Comm_free instead. I have also attached the logfiles generated while building MPICH, as described in the README, except for the file mpich-3.2.1/src/pm/hydra/tools/topo/hwloc/hwloc/config.log, as that was missing. I am using MPICH version 3.2.1, which is the latest stable release at the time of writing. Thanks in advance for your help! Best wishes, Thomas Pak _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Comm_spawn zombies have risen from the dead.
by Zhou, Hui 06 Dec '18

06 Dec '18

Hi Thomas, To summarize issue: MPICH proxy does not reap processes after its exit? I’ll look into the issue. From my peruse of current MPI standards, it does not really specify the behavior after MPI_Finalize. The dynamic process part of MPI support is weak, primary due to lack of application adoption. Regardless, I agree that the proxy creates the process should be responsible to reap its dead children. — Hui Zhou On Dec 6, 2018, at 3:53 AM, Thomas Pak via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Dear all, Does anyone have any feedback on this issue? The problem still persists in MPICH-3.3 and it seems like it is pointing to a severe flaw in how MPICH handles dynamic process creation. To reiterate, the following short program using MPICH creates zombie processes indefinitely until no more processes can be created. """ #include <stdio.h> #include <mpi.h> int main(int argc, char *argv[]) { // Initialize MPI MPI_Init(NULL, NULL); // Get parent MPI_Comm parent; MPI_Comm_get_parent(&parent); // If the process was not spawned if (parent == MPI_COMM_NULL) { puts("I was not spawned!"); // Spawn child process in loop char *cmd = argv[0]; char **cmd_argv = MPI_ARGV_NULL; int maxprocs = 1; MPI_Info info = MPI_INFO_NULL; int root = 0; MPI_Comm comm = MPI_COMM_SELF; MPI_Comm intercomm; int *array_of_errcodes = MPI_ERRCODES_IGNORE; for (;;) { MPI_Comm_spawn(cmd, cmd_argv, maxprocs, info, root, comm, &intercomm, array_of_errcodes); MPI_Comm_disconnect(&intercomm); } // If process was spawned } else { puts("I was spawned!"); MPI_Comm_disconnect(&parent); } // Finalize MPI_Finalize(); } """ Thanks in advance for your help. Best wishes, Thomas Pak On Oct 16 2018, at 4:57 pm, Thomas Pak <thomas.pak(a)maths.ox.ac.uk<mailto:[email protected]>> wrote: Dear all, My MPI application spawns a large number of MPI processes using MPI_Comm_spawn over its total lifetime. Unfortunately, I have experienced that this results in zombie processes of hydra_pmi_proxy accumulating over time. As a result, my MPI application will eventually crash because there are too many zombie processes and no new processes can be created. This issue does not seem to be new; I have found references to similar issues in at least three different places: - [mpich-discuss] MPI_Comm_Spawn causing zombies of hydra_pmi_proxy<https://lists.mpich.org/pipermail/discuss/2013-March/000599.html> (March 2013) - [GitHub issue] hydra_pmi_proxy zombie MPI_comm_spawn #1677<https://github.com/pmodels/mpich/issues/1677> (October 2016) - [Google Groups] Hanging in spawn in master-slave code and hydra_pmi_proxy zombies<https://groups.google.com/forum/#!msg/mpi4py/A9mN-2UkFf8/2gnUEtArDwAJ> (December 2016) In the first reference, a dirty fix was proposed by Silvan Brändli to simply reap all the zombie processes in a loop at the start of the function HYDU_create_process in src/pm/hydra/utils/launch/launch.c. However, I have found that this comes with its own set of problems and makes my MPI application unstable. In the GitHub issue, the problem was marked as "worksforme". However, this problem persists for me and other machines that I have tried to reproduce it on, so it clearly has not yet been resolved. In the Google Groups thread, the problem was "resolved" by avoiding spawning MPI processes entirely, which is not an option in my case. Pavan Balaji mentions in the 2013 mpi-discuss thread that this issue was a problem "once upon a time". It seems to have risen from the dead again. I have attached a short and self-contained program written in C that reproduces the problem. The program simply spawns child processes using MPI_Comm_spawn in an infinite loop, where each child process exits after writing a message to stdout. I use MPI_Comm_disconnect to disconnect the Intercommunicator, but the same problem occurs when using MPI_Comm_free instead. I have also attached the logfiles generated while building MPICH, as described in the README, except for the file mpich-3.2.1/src/pm/hydra/tools/topo/hwloc/hwloc/config.log, as that was missing. I am using MPICH version 3.2.1, which is the latest stable release at the time of writing. Thanks in advance for your help! Best wishes, Thomas Pak _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Comm_spawn zombies have risen from the dead.
by Thomas Pak 06 Dec '18

06 Dec '18

Dear all, Does anyone have any feedback on this issue? The problem still persists in MPICH-3.3 and it seems like it is pointing to a severe flaw in how MPICH handles dynamic process creation. To reiterate, the following short program using MPICH creates zombie processes indefinitely until no more processes can be created. """ #include <stdio.h> #include <mpi.h> int main(int argc, char *argv[]) { // Initialize MPI MPI_Init(NULL, NULL); // Get parent MPI_Comm parent; MPI_Comm_get_parent(&parent); // If the process was not spawned if (parent == MPI_COMM_NULL) { puts("I was not spawned!"); // Spawn child process in loop char *cmd = argv[0]; char **cmd_argv = MPI_ARGV_NULL; int maxprocs = 1; MPI_Info info = MPI_INFO_NULL; int root = 0; MPI_Comm comm = MPI_COMM_SELF; MPI_Comm intercomm; int *array_of_errcodes = MPI_ERRCODES_IGNORE; for (;;) { MPI_Comm_spawn(cmd, cmd_argv, maxprocs, info, root, comm, &intercomm, array_of_errcodes); MPI_Comm_disconnect(&intercomm); } // If process was spawned } else { puts("I was spawned!"); MPI_Comm_disconnect(&parent); } // Finalize MPI_Finalize(); } """ Thanks in advance for your help. Best wishes, Thomas Pak On Oct 16 2018, at 4:57 pm, Thomas Pak <thomas.pak(a)maths.ox.ac.uk> wrote: > > Dear all, > My MPI application spawns a large number of MPI processes using MPI_Comm_spawn over its total lifetime. Unfortunately, I have experienced that this results in zombie processes of hydra_pmi_proxy accumulating over time. As a result, my MPI application will eventually crash because there are too many zombie processes and no new processes can be created. > This issue does not seem to be new; I have found references to similar issues in at least three different places: > - [mpich-discuss] MPI_Comm_Spawn causing zombies of hydra_pmi_proxy (https://lists.mpich.org/pipermail/discuss/2013-March/000599.html) (March 2013) > - [GitHub issue] hydra_pmi_proxy zombie MPI_comm_spawn #1677 (https://github.com/pmodels/mpich/issues/1677) (October 2016) > - [Google Groups] Hanging in spawn in master-slave code and hydra_pmi_proxy zombies (https://groups.google.com/forum/#!msg/mpi4py/A9mN-2UkFf8/2gnUEtArDwAJ) (December 2016) > In the first reference, a dirty fix was proposed by Silvan Brändli to simply reap all the zombie processes in a loop at the start of the function HYDU_create_process in src/pm/hydra/utils/launch/launch.c. However, I have found that this comes with its own set of problems and makes my MPI application unstable. > > In the GitHub issue, the problem was marked as "worksforme". However, this problem persists for me and other machines that I have tried to reproduce it on, so it clearly has not yet been resolved. > In the Google Groups thread, the problem was "resolved" by avoiding spawning MPI processes entirely, which is not an option in my case. > Pavan Balaji mentions in the 2013 mpi-discuss thread that this issue was a problem "once upon a time". It seems to have risen from the dead again. > I have attached a short and self-contained program written in C that reproduces the problem. The program simply spawns child processes using MPI_Comm_spawn in an infinite loop, where each child process exits after writing a message to stdout. I use MPI_Comm_disconnect to disconnect the Intercommunicator, but the same problem occurs when using MPI_Comm_free instead. > I have also attached the logfiles generated while building MPICH, as described in the README, except for the file mpich-3.2.1/src/pm/hydra/tools/topo/hwloc/hwloc/config.log, as that was missing. I am using MPICH version 3.2.1, which is the latest stable release at the time of writing. > Thanks in advance for your help! > Best wishes, > Thomas Pak > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] symbol lookup error
by Haider Abbas 05 Dec '18

05 Dec '18

Dear *Giuseppe*, You are right, node 4 and node 6 are not working properly. when I am running pw.x on these node, it gives ./pw.x -i relax2.in >out.out & [1] 8475 mpiuser@node4:~/qe-29-11/bin$ ./pw.x: symbol lookup error: ./pw.x: undefined symbol: mpi_initialized_ I have reinstalled gfortran and mpich but still facing the same problem. Could you please further guide me how to overcome this problem. regards Haider Abbas On Wed, Dec 5, 2018 at 9:17 PM Congiu, Giuseppe via discuss < discuss(a)mpich.org> wrote: > Hello Haider, > > Are you compiling mpich and installing it in a global path accessible to > all machines or you are using local installations? The missing symbol is a > fortran symbol. The fact it is not there can mean mpich was not compiled > with fortran support while your application is a fortran code. The fact > that it works for a certain number of processes might mean some machine do > have fortran support while other machines don’t. > > Can you provide some additional information about your cluster > configuration? The MPICH version you are running and how you are > configuring it? > > --Giuseppe > > > On Dec 5, 2018, at 1:53 AM, Haider Abbas via discuss <discuss(a)mpich.org> > wrote: > > mpi_initialized_ > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] symbol lookup error
by Congiu, Giuseppe 05 Dec '18

05 Dec '18

Hello Haider, Are you compiling mpich and installing it in a global path accessible to all machines or you are using local installations? The missing symbol is a fortran symbol. The fact it is not there can mean mpich was not compiled with fortran support while your application is a fortran code. The fact that it works for a certain number of processes might mean some machine do have fortran support while other machines don’t. Can you provide some additional information about your cluster configuration? The MPICH version you are running and how you are configuring it? --Giuseppe On Dec 5, 2018, at 1:53 AM, Haider Abbas via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: mpi_initialized_

1 0

symbol lookup error
by Haider Abbas 05 Dec '18

05 Dec '18

Dear all, In continuation with my previous email, I got success in running quantum espresso on my cluster. But now I am facing another problem. with the command mpiexec -f hosts -n 10 ./pw.x -i relax2.in > out.out & QS run perfectly, but as I increase the number of core like this mpiexec -f hosts -n 30 ./pw.x -i relax2.in > out.out & it gives an error mpiuser@master:~/qe-29-11/bin$ mpiexec -f hosts -n 30 ./pw.x -i relax2.in > out.out & [4] 2131 mpiuser@master:~/qe-29-11/bin$ ./pw.x: symbol lookup error: ./pw.x: undefined symbol: mpi_initialized_ ./pw.x: symbol lookup error: ./pw.x: undefined symbol: mpi_initialized_ ./pw.x: symbol lookup error: ./pw.x: undefined symbol: mpi_initialized_ ./pw.x: symbol lookup error: ./pw.x: undefined symbol: mpi_initialized_ ./pw.x: symbol lookup error: ./pw.x: undefined symbol: mpi_initialized_ ./pw.x: symbol lookup error: ./pw.x: undefined symbol: mpi_initialized_ ./pw.x: symbol lookup error: ./pw.x: undefined symbol: mpi_initialized_ ./pw.x: symbol lookup error: ./pw.x: undefined symbol: mpi_initialized_ However I have recent hydra supported version of mpich on all the nodes I request you all to kindly help me to overcome this Problem. with regards Haider abbas Department of Physics Manav Rachna University Faridabad, Haryana, India

1 0

Re: [mpich-discuss] shared library error
by Raffenetti, Kenneth J. 04 Dec '18

04 Dec '18

If you run mpiexec -f hosts -n 40 ldd pw.x do all instances show correct linkage? On 12/3/18 11:36 PM, Haider Abbas via discuss wrote: > Dear all, > > I have prepared a 10 node cluster (each having 4 core). I have installed > GAMESS and QUANTUM ESPRESSO. Gamess is running fine, but when I run > quantum espresso with the command > > mpiexec -n 4 ./pw.x -i relax2.in <http://relax2.in> > out.out & > > I get no error and everything is fine. And the job run on all 4 core a > single node. > > But when I run the same job on the 10 node with the command > > mpiexec -f hosts -n 40 ./pw.x -i relax2.in <http://relax2.in> > out.out & > > I get the error > > > mpiuser@master:~/qe-29-11/bin$ ./pw.x: error while loading shared > libraries: liblapack.so.3: cannot open shared object file: No such file > or directory > > ./pw.x: error while loading shared libraries: liblapack.so.3: cannot > open shared object file: No such file or directory > > ./pw.x: error while loading shared libraries: liblapack.so.3: cannot > open shared object file: No such file or directory > > > > However, I have "liblapack.so.3" in the same directory from where I am > running the job. > > I sought your help. > Thanking in advance. > with regards > > Haider Abbas > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

shared library error
by Haider Abbas 04 Dec '18

04 Dec '18

Dear all, I have prepared a 10 node cluster (each having 4 core). I have installed GAMESS and QUANTUM ESPRESSO. Gamess is running fine, but when I run quantum espresso with the command mpiexec -n 4 ./pw.x -i relax2.in > out.out & I get no error and everything is fine. And the job run on all 4 core a single node. But when I run the same job on the 10 node with the command mpiexec -f hosts -n 40 ./pw.x -i relax2.in > out.out & I get the error mpiuser@master:~/qe-29-11/bin$ ./pw.x: error while loading shared libraries: liblapack.so.3: cannot open shared object file: No such file or directory ./pw.x: error while loading shared libraries: liblapack.so.3: cannot open shared object file: No such file or directory ./pw.x: error while loading shared libraries: liblapack.so.3: cannot open shared object file: No such file or directory However, I have "liblapack.so.3" in the same directory from where I am running the job. I sought your help. Thanking in advance. with regards Haider Abbas

1 0