MPI_Iprobe Error with MUMPS Solver on Multi-Nodes
Hi, I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ``` The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error. The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6] Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error stack: PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) ``` Thank you very much for your time and consideration. Best wishes, Zongze
On Mon, May 22, 2023 at 11:07 AM Zongze Yang <[email protected]> wrote:
Hi,
I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes:
Iprobe is buggy on several MPI implementations. PETSc has an option for shutting it off for this reason. I do not know how to shut it off inside MUMPS however. I would mail their mailing list to see. Thanks, Matt
```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ```
The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error.
The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6] Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error stack: PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) ```
Thank you very much for your time and consideration.
Best wishes, Zongze
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
Thanks! Zongze Matthew Knepley <[email protected]>于2023年5月23日 周二00:09写道：
On Mon, May 22, 2023 at 11:07 AM Zongze Yang <[email protected]> wrote:
Hi,
I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes:
Iprobe is buggy on several MPI implementations. PETSc has an option for shutting it off for this reason. I do not know how to shut it off inside MUMPS however. I would mail their mailing list to see.
Thanks,
Matt
```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ```
The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error.
The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6] Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error stack: PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) ```
Thank you very much for your time and consideration.
Best wishes, Zongze
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- Best wishes, Zongze
If I may add to the discussion, it may be that you are going OOM since you are trying to factorize a 3 million dofs problem, this problem goes undetected and then fails at a later stage Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang <[email protected]> ha scritto:
Thanks!
Zongze
Matthew Knepley <[email protected]>于2023年5月23日 周二00:09写道：
On Mon, May 22, 2023 at 11:07 AM Zongze Yang <[email protected]> wrote:
Hi,
I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes:
Iprobe is buggy on several MPI implementations. PETSc has an option for shutting it off for this reason. I do not know how to shut it off inside MUMPS however. I would mail their mailing list to see.
Thanks,
Matt
```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ```
The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error.
The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6] Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error stack: PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) ```
Thank you very much for your time and consideration.
Best wishes, Zongze
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- Best wishes, Zongze
-- Stefano
On Tue, 23 May 2023 at 05:31, Stefano Zampini <[email protected]> wrote:
If I may add to the discussion, it may be that you are going OOM since you are trying to factorize a 3 million dofs problem, this problem goes undetected and then fails at a later stage
Thank you for your comment. I ran the problem with 90 processes distributed across three nodes, each equipped with 500G of memory. If this amount of memory is sufficient for solving the matrix with approximately 3 million degrees of freedom? Thanks! Zongze Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang <[email protected]>
ha scritto:
Thanks!
Zongze
Matthew Knepley <[email protected]>于2023年5月23日 周二00:09写道：
On Mon, May 22, 2023 at 11:07 AM Zongze Yang <[email protected]> wrote:
Hi,
I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes:
Iprobe is buggy on several MPI implementations. PETSc has an option for shutting it off for this reason. I do not know how to shut it off inside MUMPS however. I would mail their mailing list to see.
Thanks,
Matt
```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ```
The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error.
The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6] Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error stack: PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) ```
Thank you very much for your time and consideration.
Best wishes, Zongze
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- Best wishes, Zongze
-- Stefano
I have an additional question to ask: Is it possible for the SuperLU_DIST library to encounter the same MPI problem (PMPI_Iprobe failed) as MUMPS? Best wishes, Zongze On Tue, 23 May 2023 at 10:41, Zongze Yang <[email protected]> wrote:
On Tue, 23 May 2023 at 05:31, Stefano Zampini <[email protected]> wrote:
If I may add to the discussion, it may be that you are going OOM since you are trying to factorize a 3 million dofs problem, this problem goes undetected and then fails at a later stage
Thank you for your comment. I ran the problem with 90 processes distributed across three nodes, each equipped with 500G of memory. If this amount of memory is sufficient for solving the matrix with approximately 3 million degrees of freedom?
Thanks! Zongze
Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang <[email protected]>
ha scritto:
Thanks!
Zongze
Matthew Knepley <[email protected]>于2023年5月23日 周二00:09写道：
On Mon, May 22, 2023 at 11:07 AM Zongze Yang <[email protected]> wrote:
Hi,
I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes:
Iprobe is buggy on several MPI implementations. PETSc has an option for shutting it off for this reason. I do not know how to shut it off inside MUMPS however. I would mail their mailing list to see.
Thanks,
Matt
```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ```
The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error.
The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6] Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error stack: PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) ```
Thank you very much for your time and consideration.
Best wishes, Zongze
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- Best wishes, Zongze
-- Stefano
On Mon, May 22, 2023 at 10:46 PM Zongze Yang <[email protected]> wrote:
I have an additional question to ask: Is it possible for the SuperLU_DIST library to encounter the same MPI problem (PMPI_Iprobe failed) as MUMPS?
I do not know if they use that function. But it is easy to try it out, so I would. Thanks, Matt
Best wishes, Zongze
On Tue, 23 May 2023 at 10:41, Zongze Yang <[email protected]> wrote:
On Tue, 23 May 2023 at 05:31, Stefano Zampini <[email protected]> wrote:
If I may add to the discussion, it may be that you are going OOM since you are trying to factorize a 3 million dofs problem, this problem goes undetected and then fails at a later stage
Thank you for your comment. I ran the problem with 90 processes distributed across three nodes, each equipped with 500G of memory. If this amount of memory is sufficient for solving the matrix with approximately 3 million degrees of freedom?
Thanks! Zongze
Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang <
[email protected]> ha scritto:
Thanks!
Zongze
Matthew Knepley <[email protected]>于2023年5月23日 周二00:09写道：
On Mon, May 22, 2023 at 11:07 AM Zongze Yang <[email protected]> wrote:
Hi,
I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes:
Iprobe is buggy on several MPI implementations. PETSc has an option for shutting it off for this reason. I do not know how to shut it off inside MUMPS however. I would mail their mailing list to see.
Thanks,
Matt
```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ```
The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error.
The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6] Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error stack: PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) ```
Thank you very much for your time and consideration.
Best wishes, Zongze
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- Best wishes, Zongze
-- Stefano
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
Thank you for your suggestion. I solved the problem with SuperLU_DIST, and it works well. Best wishes, Zongze On Tue, 23 May 2023 at 18:00, Matthew Knepley <[email protected]> wrote:
On Mon, May 22, 2023 at 10:46 PM Zongze Yang <[email protected]> wrote:
I have an additional question to ask: Is it possible for the SuperLU_DIST library to encounter the same MPI problem (PMPI_Iprobe failed) as MUMPS?
I do not know if they use that function. But it is easy to try it out, so I would.
Thanks,
Matt
Best wishes, Zongze
On Tue, 23 May 2023 at 10:41, Zongze Yang <[email protected]> wrote:
On Tue, 23 May 2023 at 05:31, Stefano Zampini <[email protected]> wrote:
If I may add to the discussion, it may be that you are going OOM since you are trying to factorize a 3 million dofs problem, this problem goes undetected and then fails at a later stage
Thank you for your comment. I ran the problem with 90 processes distributed across three nodes, each equipped with 500G of memory. If this amount of memory is sufficient for solving the matrix with approximately 3 million degrees of freedom?
Thanks! Zongze
Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang <
[email protected]> ha scritto:
Thanks!
Zongze
Matthew Knepley <[email protected]>于2023年5月23日 周二00:09写道：
On Mon, May 22, 2023 at 11:07 AM Zongze Yang <[email protected]> wrote:
Hi,
I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes:
Iprobe is buggy on several MPI implementations. PETSc has an option for shutting it off for this reason. I do not know how to shut it off inside MUMPS however. I would mail their mailing list to see.
Thanks,
Matt
```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ```
The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error.
The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6] Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error stack: PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) ```
Thank you very much for your time and consideration.
Best wishes, Zongze
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- Best wishes, Zongze
-- Stefano
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
On Tue, 23 May 2023 at 19:51, Zongze Yang <[email protected]> wrote:
Thank you for your suggestion. I solved the problem with SuperLU_DIST, and it works well.
This is solved with four nodes, each equipped with 500G of memory. Best wishes, Zongze Best wishes,
Zongze
On Tue, 23 May 2023 at 18:00, Matthew Knepley <[email protected]> wrote:
On Mon, May 22, 2023 at 10:46 PM Zongze Yang <[email protected]> wrote:
I have an additional question to ask: Is it possible for the SuperLU_DIST library to encounter the same MPI problem (PMPI_Iprobe failed) as MUMPS?
I do not know if they use that function. But it is easy to try it out, so I would.
Thanks,
Matt
Best wishes, Zongze
On Tue, 23 May 2023 at 10:41, Zongze Yang <[email protected]> wrote:
On Tue, 23 May 2023 at 05:31, Stefano Zampini < [email protected]> wrote:
If I may add to the discussion, it may be that you are going OOM since you are trying to factorize a 3 million dofs problem, this problem goes undetected and then fails at a later stage
Thank you for your comment. I ran the problem with 90 processes distributed across three nodes, each equipped with 500G of memory. If this amount of memory is sufficient for solving the matrix with approximately 3 million degrees of freedom?
Thanks! Zongze
Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang <
[email protected]> ha scritto:
Thanks!
Zongze
Matthew Knepley <[email protected]>于2023年5月23日 周二00:09写道：
On Mon, May 22, 2023 at 11:07 AM Zongze Yang <[email protected]> wrote:
Hi,
I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes:
Iprobe is buggy on several MPI implementations. PETSc has an option for shutting it off for this reason. I do not know how to shut it off inside MUMPS however. I would mail their mailing list to see.
Thanks,
Matt
```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ```
The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error.
The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6] Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error stack: PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) ```
Thank you very much for your time and consideration.
Best wishes, Zongze
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- Best wishes, Zongze
-- Stefano
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
On Mon, May 22, 2023 at 10:42 PM Zongze Yang <[email protected]> wrote:
On Tue, 23 May 2023 at 05:31, Stefano Zampini <[email protected]> wrote:
If I may add to the discussion, it may be that you are going OOM since you are trying to factorize a 3 million dofs problem, this problem goes undetected and then fails at a later stage
Thank you for your comment. I ran the problem with 90 processes distributed across three nodes, each equipped with 500G of memory. If this amount of memory is sufficient for solving the matrix with approximately 3 million degrees of freedom?
It really depends on the fill. Suppose that you get 1% fill, then (3e6)^2 * 0.01 * 8 = 1e12 B and you have 1.5e12 B, so I could easily see running out of memory. Thanks, Matt
Thanks! Zongze
Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang <[email protected]>
ha scritto:
Thanks!
Zongze
Matthew Knepley <[email protected]>于2023年5月23日 周二00:09写道：
On Mon, May 22, 2023 at 11:07 AM Zongze Yang <[email protected]> wrote:
Hi,
I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes:
Iprobe is buggy on several MPI implementations. PETSc has an option for shutting it off for this reason. I do not know how to shut it off inside MUMPS however. I would mail their mailing list to see.
Thanks,
Matt
```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ```
The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error.
The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6] Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error stack: PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) ```
Thank you very much for your time and consideration.
Best wishes, Zongze
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- Best wishes, Zongze
-- Stefano
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
If i may, you can use the command line option "-mat_mumps_icntl_4 2" MUMPS then gives infomations about the factorization step, such as the estimated needed memory. Best regards, Yann Le 5/23/2023 à 11:59 AM, Matthew Knepley a écrit :
On Mon, May 22, 2023 at 10:42 PM Zongze Yang <[email protected] <mailto:[email protected]>> wrote:
On Tue, 23 May 2023 at 05:31, Stefano Zampini <[email protected] <mailto:[email protected]>> wrote:
If I may add to the discussion, it may be that you are going OOM since you are trying to factorize a 3 million dofs problem, this problem goes undetected and then fails at a later stage
Thank you for your comment. I ran the problem with 90 processes distributed across three nodes, each equipped with 500G of memory. If this amount of memory is sufficient for solving the matrix with approximately 3 million degrees of freedom?
It really depends on the fill. Suppose that you get 1% fill, then
(3e6)^2 * 0.01 * 8 = 1e12 B
and you have 1.5e12 B, so I could easily see running out of memory.
Thanks,
Matt
Thanks! Zongze
Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang <[email protected] <mailto:[email protected]>> ha scritto:
Thanks!
Zongze
Matthew Knepley <[email protected] <mailto:[email protected]>>于2023年5月23日 周二00:09写道：
On Mon, May 22, 2023 at 11:07 AM Zongze Yang <[email protected] <mailto:[email protected]>> wrote:
Hi,
I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes:
Iprobe is buggy on several MPI implementations. PETSc has an option for shutting it off for this reason. I do not know how to shut it off inside MUMPS however. I would mail their mailing list to see.
Thanks,
Matt
```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ```
The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error.
The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6] Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error stack: PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) ```
Thank you very much for your time and consideration.
Best wishes, Zongze
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- Best wishes, Zongze
-- Stefano
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
On Tue, 23 May 2023 at 20:09, Yann Jobic <[email protected]> wrote:
If i may, you can use the command line option "-mat_mumps_icntl_4 2" MUMPS then gives infomations about the factorization step, such as the estimated needed memory.
Thank you for your suggestion!
Best wishes, Zongze Best regards,
Yann
Le 5/23/2023 à 11:59 AM, Matthew Knepley a écrit :
On Mon, May 22, 2023 at 10:42 PM Zongze Yang <[email protected] <mailto:[email protected]>> wrote:
On Tue, 23 May 2023 at 05:31, Stefano Zampini <[email protected] <mailto:[email protected]>> wrote:
If I may add to the discussion, it may be that you are going OOM since you are trying to factorize a 3 million dofs problem, this problem goes undetected and then fails at a later stage
Thank you for your comment. I ran the problem with 90 processes distributed across three nodes, each equipped with 500G of memory. If this amount of memory is sufficient for solving the matrix with approximately 3 million degrees of freedom?
It really depends on the fill. Suppose that you get 1% fill, then
(3e6)^2 * 0.01 * 8 = 1e12 B
and you have 1.5e12 B, so I could easily see running out of memory.
Thanks,
Matt
Thanks! Zongze
Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang <[email protected] <mailto:[email protected]>> ha scritto:
Thanks!
Zongze
Matthew Knepley <[email protected] <mailto:[email protected]>>于2023年5月23日 周二00:09写道：
On Mon, May 22, 2023 at 11:07 AM Zongze Yang <[email protected] <mailto:[email protected]>> wrote:
Hi,
I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes:
Iprobe is buggy on several MPI implementations. PETSc has an option for shutting it off for this reason. I do not know how to shut it off inside MUMPS however. I would mail their mailing list to see.
Thanks,
Matt
```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed
(ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error)
Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ```
The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error.
The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000026, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed
(ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error)
Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542]
/nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36]
Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed
(ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error)
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_b_+0x1463) [0x7f60738831a1]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_driver_+0x6969) [0x7f60738446c9]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_+0x2d83) [0x7f60738bf9cf]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_f77_+0x178c) [0x7f60738c33bc]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_c+0x8f8) [0x7f60738baacb]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x894560) [0x7f6077297560]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(MatLUFactorNumeric+0x32e) [0x7f60773bb1e6]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0xf51665) [0x7f6077954665]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(PCSetUp+0x64b) [0x7f60779c77e0]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSetUp+0xfb6) [0x7f6077ac2d53]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x10c1c28) [0x7f6077ac4c28]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(KSPSolve+0x13) [0x7f6077ac8070]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(+0x11249df) [0x7f6077b279df]
/nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/petsc-develop-5wrc3y6lyelr3iyrlm3sr2jlh2wxif3k/lib/libpetsc.so.3.019(SNESSolve+0x10df) [0x7f6077b676c6]
Abort(1) on node 60: Internal error Abort(1007265423) on node 65 (rank 65 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff4d82827c, status=0x7fff4d8284b0) failed MPID_Iprobe(244)..............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed
(ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error)
Abort(941205135) on node 32 (rank 32 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..............: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc4000017, flag=0x7fff715ba3fc, status=0x7fff715ba630) failed MPID_Iprobe(240)..............: MPIDI_iprobe_safe(108)........: MPIDI_iprobe_unsafe(35).......: MPIDI_OFI_do_iprobe(69).......: MPIDI_OFI_handle_cq_error(949): OFI poll failed
(ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error)
Abort(470941839) on node 75 (rank 75 in comm 0): Fatal error in PMPI_Test: Other MPI error, error
stack:
PMPI_Test(188)................: MPI_Test(request=0x7efe31e03014, flag=0x7ffea65d673c, status=0x7ffea65d6760) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed
(ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error)
Abort(805946511) on node 31 (rank 31 in comm 256): Fatal error in PMPI_Probe: Other MPI error, error
stack:
PMPI_Probe(118)...............: MPI_Probe(src=MPI_ANY_SOURCE, tag=7, comm=0xc4000015, status=0x7fff9538b7a0) failed MPID_Probe(159)...............: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed
(ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error)
Abort(1179791) on node 73 (rank 73 in comm 0): Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(188)................: MPI_Test(request=0x5b638d4, flag=0x7ffd755119cc, status=0x7ffd755121b0) failed MPIR_Test(73).................: MPIR_Test_state(33)...........: progress_test(100)............: MPIDI_OFI_handle_cq_error(949): OFI poll failed
(ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error)
```
Thank you very much for your time and consideration.
Best wishes, Zongze
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- Best wishes, Zongze
-- Stefano
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
participants (4)
-
Matthew Knepley -
Stefano Zampini -
Yann Jobic -
Zongze Yang