I'm not talking about timings, I am talking about seeing where a code dies.

When codes die within the MPI implementation is basically impossible to provide a backtrace from PETSc, but the MPI implementation may offer some verbose messaging. If this happens quite early in the code (and I presume here we are failing somewhere in the 1-to-all pattern when distributing the plex data structure) running with -log_trace may get you some insight on where this is happening (provided that the offending PETSc routine calls LogEventBegin/End) In my experience, the sad story is that when there are software vendor failures with a large number of processes, I haven't found anything more efficient than a few bisection steps with good old printfs....

...

As a side note, one would expect that this kind of behavior is consistent across the library. But in fact, it isn’t. If you do something like -sub_ksp_view_pmat file with, for example, DD preconditioners, you end up with a single file with all local matrices scrambled in. It would be nice to have the same logic, i.e., file.%p, and be able to filter ranks that actually print to file like you suggest. These kind of options are less critical than -info so I can just insert them in the option database after initialization, if rank == view_this_rank PetscOptionsInsertString(), but doing that using command line arguments would be easier and more elegant, IMO.

Thanks, Pierre

Barry

On Apr 22, 2021, at 11:14 AM, Mark Adams <[email protected]> wrote:

OK, so -info out.%p would do what I asked. Alas, what I really need it to reduce the output with 128K processes. Thanks,

On Thu, Apr 22, 2021 at 11:54 AM Pierre Jolivet <[email protected]> wrote:

...

On 22 Apr 2021, at 5:33 PM, Jacob Faibussowitsch <[email protected]> wrote:

Not out of the box no. There is -info ::self which makes info only print from calls made on PETSC_COMM_SELF, but I don’t think that’s what you want. Best advice would be to print to a file and grep the file for any lines starting with the rank number you want.

There is no need for grep, IMHO, just do -info file and then open file.%p

Thanks, Pierre

Best regards,

Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch)

On Apr 22, 2021, at 10:16, Mark Adams <[email protected]> wrote:

Can I make -info only print from one process? Mark

-- Stefano