I'm not talking about timings, I am talking about seeing where a code dies.

When codes die within the MPI implementation is basically impossible to provide a backtrace from PETSc, but the MPI implementation may offer some verbose messaging.

If this happens quite early in the code (and I presume here we are failing somewhere in the 1-to-all pattern when distributing the plex data structure) running with -log_trace may get you some insight on where this is happening  (provided that the offending PETSc routine calls LogEventBegin/End)

In my experience, the sad story is that when there are software vendor failures with a large number of processes, I haven't found anything more efficient than a few bisection steps with good old printfs.... 

 

 

As a side note, one would expect that this kind of behavior is consistent across the library.

But in fact, it isn’t. If you do something like -sub_ksp_view_pmat file with, for example, DD preconditioners, you end up with a single file with all local matrices scrambled in.

It would be nice to have the same logic, i.e., file.%p, and be able to filter ranks that actually print to file like you suggest.

These kind of options are less critical than -info so I can just insert them in the option database after initialization, if rank == view_this_rank PetscOptionsInsertString(), but doing that using command line arguments would be easier and more elegant, IMO. 

Thanks,

Pierre

  Barry

On Apr 22, 2021, at 11:14 AM, Mark Adams <mfadams@lbl.gov> wrote:

OK, so -info out.%p would do what I asked. Alas, what I really need it to reduce the output with 128K processes.

Thanks,

On Thu, Apr 22, 2021 at 11:54 AM Pierre Jolivet <pierre@joliv.et> wrote:

On 22 Apr 2021, at 5:33 PM, Jacob Faibussowitsch <jacob.fai@gmail.com> wrote:

Not out of the box no. There is -info ::self which makes info only print from calls made on PETSC_COMM_SELF, but I don’t think that’s what you want. Best advice would be to print to a file and grep the file for any lines starting with the rank number you want.

There is no need for grep, IMHO, just do -info file and then open file.%p

Thanks,

Pierre

Best regards,

Jacob Faibussowitsch
(Jacob Fai - booss - oh - vitch)

On Apr 22, 2021, at 10:16, Mark Adams <mfadams@lbl.gov> wrote:

Can I make -info only print from one process?

Mark

--

Stefano