OK. After help from several forums, I think I understand the cause of the problem. As Jeff said, it has nothing to do with MPI. I do need 2 processes to trigger this issue though. Hui's comment also helps. Linux allows over committing. See here <https://www.kernel.org/doc/Documentation/vm/overcommit-accounting> and here <https://www.etalabs.net/overcommit.html>. In my case, say the machine has 32GB RAM, 2GB is already used and each MPI process is trying to allocate 8GB at a time. Then this happens: Initial memory usage: 2GB After the first memory allocation: 18GB In the second memory allocation, each MPI process thinks it has enough space for 8GB because of over committing, and writes to it. But that requires 34GB, exceeding RAM size. So the out of memory killer on Linux kills one of the MPI process (sends a SIGKILL signal), and the other MPI process receives a SIGTERM signal. This also explains my questions above. The reason this problem doesn't happen on Windows is Windows doesn't allow over commit <https://superuser.com/questions/1194263/will-microsoft-windows-10-overcommit-memory>. I don't know about Mac. Thanks again everyone. Best regards, Zhen On Wed, Apr 3, 2019 at 1:23 PM Joachim Protze via discuss <[email protected]> wrote:
Jeff,
as I understand the original question, this is the intention of the statement: reproducibly trigger an allocation exception, which should be caught by the try-catch block.
Hui,
As you will probably experience, your intended behavior will not be reached with replacing the vector resize by allocation with new:
std::vector<double*> a(100); ... a[i] = new double[1000000000];
The reason is, that new allocates virtual memory. You will see a segfault, when you try to initialize the whole array and request physical memory for the array. The vector resize operation allocates the memory and immediately initializes the memory. Therefore not the allocation itself fails, but finally the access to the memory during initialization. I have no explanation why you can see the bad_alloc when executing it only on one rank.
Best Joachim
On 4/3/19 6:51 PM, Jeff Hammond via discuss wrote:
This error has nothing to do with MPI.
These two lines are the problem:
std::vector<std::vector<double> > a(100); a[i].resize(1000000000);
You are trying to allocate 1e9 std::vector<double>, which I assume is many gigabytes of memory.
Jeff
On Wed, Apr 3, 2019 at 8:58 AM Zhou, Hui via discuss <[email protected] <mailto:[email protected]>> wrote:
I just tested on my Mac, the system keep popping out message asking
me to force kill some application to free up memory, but the code never crashed, just stalled. So I guess the out-of-memory behavior is operating system dependent. My guess is when you have multiple process competing for memory, it may cause some race issue in the os that the OS returns a valid address but discover out-of memory later because the memory is stolen by another process.
— Hui Zhou T: 630-252-3430
On Apr 3, 2019, at 8:42 AM, Zhen Wang via discuss <[email protected]
<mailto:[email protected]>> wrote:
Hi,
I have difficulty catching std::bac_alloc in an MPI environment. The
code is attached. I'm uisng gcc 6.3 on SUSE Linux Enterprise Server 11 (x86_64). mpich is built from source. The commands are as follows:
Build g++ -I<mpich-3.3-opt/include> -L<mpich-3.3-opt/lib> -lmpi memory.cpp
Run <mpich-3.3-opt/bin/mpiexec> -n 2 a.out
Output 0 0 1 1
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 16067 RUNNING AT <machine name> = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions
If I uncomment the line //if (rank == 0), i.e., only rank 0 allocates memory, I'm able to catch bad_alloc as I expected. It seems that I am misunderstanding something. Could you please help? Thanks a lot.
Best regards, Zhen <memory.cpp>_______________________________________________ discuss mailing list [email protected] <mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list [email protected] <mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
-- Jeff Hammond [email protected] <mailto:[email protected]> http://jeffhammond.github.io/
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
-- Dipl.-Inf. Joachim Protze
IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 [email protected] www.itc.rwth-aachen.de
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss