Thanks. I had further discussion on this on github: https://urldefense.us/v3/__https://github.com/pmodels/mpich/discussions/7052... I’m reposting my last response there as I need some answers. We got the confirmation from the customer that there was a security application which was somehow killing the application as the application(and it's executables) were not whitelisted. I have couple of questions: 1. Is the latest stable release of mpi4.2.2 stable against such instances? Is single machine run still require TCP ports or is it done using shared memory? 2. Is there anything that we can do while configuring mpich 3.2.1(the current version we are using) to make the application more robust against such instances? As I said it's a single machine run, so I'm still not sure why TCP ports are involved. Thanks Tarun From: Raffenetti, Ken <[email protected]> Sent: Sunday, July 7, 2024 5:24 AM To: [email protected] Cc: Kumar, Tarun (DI SW ICS DVT RD QSCE) <[email protected]> Subject: Re: [mpich-discuss] Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... Are you able to update the MPICH version? MPICH 3.2.1 was released in 2017 and is no longer actively supported/maintained. Ken From: "kumar.tarun--- via discuss" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Saturday, July 6, 2024 at 3:49 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [mpich-discuss] Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... Hi, We are hitting following assertion: Assertion failed in file .. . nemesis/netmod/tcp/socksm. c at line 600: hdr. pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || .. . I looked at the assert and it looks like this in file …/nemesis/netmod/tcp/socksm. c ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, We are hitting following assertion: Assertion failed in file ...nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... I looked at the assert and it looks like this in file …/nemesis/netmod/tcp/socksm.c MPIU_Assert(hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO); We have tried multiple cores/partitions from 2 to 8 and the behaviour is same. Also a process is aborted and a message appears to suggest that. Mostly it's process 0 which is aborted but I have seen other processes as well reporting the crash. We are using mpich-3.2.1. I'm trying to understand possible causes for this error? I have explored the forum and no possible causes, like machine going out of memory etc are applicable here. Please suggest. Are there any debug/log/trace options I can use with mpiexec to further root cause? Regards Tarun