If you want to read large files, you should really use MPI-I/O, not stdin. We support reading from stdin for users who migrate from sequential programs to parallel programming, but that's not meant to be the real mechanism to read files. I'm not supportive of trying to modify mpich for such use cases. -- Pavan

On Jun 5, 2016, at 11:18 AM, Seo, Sangmin <[email protected]> wrote:

Hello Salvatore,

The current MPICH uses pipe to redirect stdin to processes, but pipe has a limited capacity (e.g., 64KB in Linux. Please refer to ​http://man7.org/linux/man-pages/man7/pipe.7.html). Therefore, if we redirect a file, which is larger than the pipe capacity, to stdin, the current MPICH implementation is not handling it properly because write() does not succeed to write a large data (> the pipe capacity) to the pipe for stdin.

I created a ticket for this issue (https://trac.mpich.org/projects/mpich/ticket/2336), but I'd recommend you to use file I/O in order to handle a large file instead of using stdin redirection.

— Sangmin

...

On Jun 1, 2016, at 4:37 AM, Filippone, Salvatore <[email protected]> wrote:

Hello, I have found something I do not understand with mpich 3.2.0. The problem started with an application, but to demonstrate it I have written a very simple-minded program which I am including here. The program simply reads lines from standard input and reprints them on standard output from a second process.

Now if I redirect stdin from a small(ish) file it works as expected: ====================================================================== [sfilippo@localhost MPI_cat]$ mpicc -v mpicc for MPICH version 3.2 Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/opt/gnu/5.3.0/libexec/gcc/x86_64-unknown-linux- gnu/5.3.0/lto-wrapper Target: x86_64-unknown-linux-gnu Configured with: ../gcc-5.3.0/configure --prefix=/opt/gnu/5.3.0 -- enable-languages=c,c++,fortran --with- gmp=/usr/local/travel/GCC/BUILDS/gmp --with- mpfr=/usr/local/travel/GCC/BUILDS/mpfr --with- mpc=/usr/local/travel/GCC/BUILDS/mpc --with- isl=/usr/local/travel/GCC/BUILDS/isl Thread model: posix gcc version 5.3.0 (GCC) [sfilippo@localhost MPI_cat]$ mpicc -o mpicat mpicat.c [sfilippo@localhost MPI_cat]$ head mpicat.c | mpirun -np 2 ./mpicat #include <stdlib.h> #include <stdio.h> #include "mpi.h" #define LNSZ 129 #define TAGI 234 #define TAGC 567 int main(int argc, char *argv[] ) { char linebuf[LNSZ]; int ierr, iam, np, itmp ; [sfilippo@localhost MPI_cat]$ ====================================================================== However if I use a long(ish) file (such as a configure script) I get either no output or an error. On another machine with GNU 4.9.3 I always get the silent failure. ====================================================================== [sfilippo@localhost MPI_cat]$ mpirun -np 2 ./mpicat < configure [sfilippo@localhost MPI_cat]$ mpirun -np 2 ./mpicat < configure [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-3.2/src/pm/hydra/pm/pmiserv/pmip_cb.c:937): process reading stdin too slowly; can't keep up [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-3.2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich- 3.2/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [sfilippo@localhost MPI_cat]$ =======================================================================

Am I missing something? Is this supposed to work? I could find nothing relevant in the FAQ

Thanks a lot Salvatore -- Salvatore Filippone, PhD Centre for Computational Engineering Sciences, School of Aerospace, Transport and Manufacturing, Bldg 52 Cranfield University Cranfield MK43 0AL United Kingdom tel: +44(0) 1234 758361

<mpicat.c>_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss