April 2015 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [OMPI users] potential bug with MPI_Win_fence() in openmpi-1.8.4
by Satish Balay 30 Apr '15

30 Apr '15

Great! Thanks for checking. Satish On Thu, 30 Apr 2015, George Bosilca wrote: > I went over the code and in fact I think it is correct as is. The length is > for the local representation, which indeed uses pointers to datatype > structures. On the opposite, the total_pack_size represents the amount of > space we would need to store the data in a format that can be sent to > another peer, in which case handling pointers is pointless and we fall back > to int. > > However, I think we are counting twice the space needed for predefined > data. I'll push a patch shortly. > > George. > > > On Thu, Apr 30, 2015 at 3:33 PM, George Bosilca <bosilca(a)icl.utk.edu> wrote: > > > In the packed representation we store not MPI_Datatypes but a handcrafted > > id for each one. The 2 codes should have been in sync. I'm looking at > > another issue right now, and I'll come back to this one right after. > > > > Thanks for paying attention to the code. > > George. > > > > On Thu, Apr 30, 2015 at 3:13 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > >> Thanks for checking and getting a more appropriate fix in. > >> > >> I've just tried this out - and the PETSc test code runs fine with it. > >> > >> BTW: There is one inconsistancy in ompi/datatype/ompi_datatype_args.c > >> [that I noticed] - that you might want to check. > >> Perhaps the second line should be "(DC) * sizeof(MPI_Datatype)"? > >> > >> >>>>>>>>> > >> int length = sizeof(ompi_datatype_args_t) + (IC) * sizeof(int) + \ > >> (AC) * sizeof(OPAL_PTRDIFF_TYPE) + (DC) * > >> sizeof(MPI_Datatype); \ > >> > >> > >> pArgs->total_pack_size = (4 + (IC)) * sizeof(int) + \ > >> (AC) * sizeof(OPAL_PTRDIFF_TYPE) + (DC) * sizeof(int); \ > >> <<<<<<<<<<< > >> > >> Satish > >> > >> > >> On Thu, 30 Apr 2015, Matthew Knepley wrote: > >> > >> > On Fri, May 1, 2015 at 4:55 AM, Jeff Squyres (jsquyres) < > >> jsquyres(a)cisco.com> > >> > wrote: > >> > > >> > > Thank you! > >> > > > >> > > George reviewed your patch and adjusted it a bit. We applied it to > >> master > >> > > and it's pending to the release series (v1.8.x). > >> > > > >> > > >> > Was this identified by IBM? > >> > > >> > > >> > > >> https://github.com/open-mpi/ompi/commit/015d3f56cf749ee5ad9ea4428d2f5da72f9… > >> > > >> > Matt > >> > > >> > > >> > > Would you mind testing a nightly master snapshot? It should be in > >> > > tonight's build: > >> > > > >> > > http://www.open-mpi.org/nightly/master/ > >> > > > >> > > > >> > > > >> > > > On Apr 30, 2015, at 12:50 AM, Satish Balay <balay(a)mcs.anl.gov> > >> wrote: > >> > > > > >> > > > OpenMPI developers, > >> > > > > >> > > > We've had issues (memory errors) with OpenMPI - and code in PETSc > >> > > > library that uses MPI_Win_fence(). > >> > > > > >> > > > Vagrind shows memory corruption deep inside OpenMPI function stack. > >> > > > > >> > > > I'm attaching a potential patch that appears to fix this issue for > >> us. > >> > > > [the corresponding valgrind trace is listed in the patch header] > >> > > > > >> > > > Perhaps there is a more appropriate fix for this memory corruption. > >> Could > >> > > > you check on this? > >> > > > > >> > > > [Sorry I don't have a pure MPI test code to demonstrate this error - > >> > > > but a PETSc test example code consistantly reproduces this issue] > >> > > > > >> > > > Thanks, > >> > > > Satish<openmpi-1.8.4.patch> > >> > > > >> > > > >> > > -- > >> > > Jeff Squyres > >> > > jsquyres(a)cisco.com > >> > > For corporate legal information go to: > >> > > http://www.cisco.com/web/about/doing_business/legal/cri/ > >> > > > >> > > > >> > > >> > > >> > > >> > >> _______________________________________________ > >> users mailing list > >> users(a)open-mpi.org > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >> Link to this post: > >> http://www.open-mpi.org/community/lists/users/2015/04/26823.php > >> > > > > >

1 0

Re: [petsc-dev] [OMPI users] potential bug with MPI_Win_fence() in openmpi-1.8.4
by George Bosilca 30 Apr '15

30 Apr '15

I went over the code and in fact I think it is correct as is. The length is for the local representation, which indeed uses pointers to datatype structures. On the opposite, the total_pack_size represents the amount of space we would need to store the data in a format that can be sent to another peer, in which case handling pointers is pointless and we fall back to int. However, I think we are counting twice the space needed for predefined data. I'll push a patch shortly. George. On Thu, Apr 30, 2015 at 3:33 PM, George Bosilca <bosilca(a)icl.utk.edu> wrote: > In the packed representation we store not MPI_Datatypes but a handcrafted > id for each one. The 2 codes should have been in sync. I'm looking at > another issue right now, and I'll come back to this one right after. > > Thanks for paying attention to the code. > George. > > On Thu, Apr 30, 2015 at 3:13 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > >> Thanks for checking and getting a more appropriate fix in. >> >> I've just tried this out - and the PETSc test code runs fine with it. >> >> BTW: There is one inconsistancy in ompi/datatype/ompi_datatype_args.c >> [that I noticed] - that you might want to check. >> Perhaps the second line should be "(DC) * sizeof(MPI_Datatype)"? >> >> >>>>>>>>> >> int length = sizeof(ompi_datatype_args_t) + (IC) * sizeof(int) + \ >> (AC) * sizeof(OPAL_PTRDIFF_TYPE) + (DC) * >> sizeof(MPI_Datatype); \ >> >> >> pArgs->total_pack_size = (4 + (IC)) * sizeof(int) + \ >> (AC) * sizeof(OPAL_PTRDIFF_TYPE) + (DC) * sizeof(int); \ >> <<<<<<<<<<< >> >> Satish >> >> >> On Thu, 30 Apr 2015, Matthew Knepley wrote: >> >> > On Fri, May 1, 2015 at 4:55 AM, Jeff Squyres (jsquyres) < >> jsquyres(a)cisco.com> >> > wrote: >> > >> > > Thank you! >> > > >> > > George reviewed your patch and adjusted it a bit. We applied it to >> master >> > > and it's pending to the release series (v1.8.x). >> > > >> > >> > Was this identified by IBM? >> > >> > >> > >> https://github.com/open-mpi/ompi/commit/015d3f56cf749ee5ad9ea4428d2f5da72f9… >> > >> > Matt >> > >> > >> > > Would you mind testing a nightly master snapshot? It should be in >> > > tonight's build: >> > > >> > > http://www.open-mpi.org/nightly/master/ >> > > >> > > >> > > >> > > > On Apr 30, 2015, at 12:50 AM, Satish Balay <balay(a)mcs.anl.gov> >> wrote: >> > > > >> > > > OpenMPI developers, >> > > > >> > > > We've had issues (memory errors) with OpenMPI - and code in PETSc >> > > > library that uses MPI_Win_fence(). >> > > > >> > > > Vagrind shows memory corruption deep inside OpenMPI function stack. >> > > > >> > > > I'm attaching a potential patch that appears to fix this issue for >> us. >> > > > [the corresponding valgrind trace is listed in the patch header] >> > > > >> > > > Perhaps there is a more appropriate fix for this memory corruption. >> Could >> > > > you check on this? >> > > > >> > > > [Sorry I don't have a pure MPI test code to demonstrate this error - >> > > > but a PETSc test example code consistantly reproduces this issue] >> > > > >> > > > Thanks, >> > > > Satish<openmpi-1.8.4.patch> >> > > >> > > >> > > -- >> > > Jeff Squyres >> > > jsquyres(a)cisco.com >> > > For corporate legal information go to: >> > > http://www.cisco.com/web/about/doing_business/legal/cri/ >> > > >> > > >> > >> > >> > >> >> _______________________________________________ >> users mailing list >> users(a)open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2015/04/26823.php >> > >

1 0

Re: [petsc-dev] [OMPI users] potential bug with MPI_Win_fence() in openmpi-1.8.4
by Jeff Squyres (jsquyres) 30 Apr '15

30 Apr '15

Per Satish's last mail (http://www.open-mpi.org/community/lists/users/2015/04/26823.php) George is looking at a followup issue... > On Apr 30, 2015, at 2:57 PM, Ralph Castain <rhc(a)open-mpi.org> wrote: > > Thanks! The patch wasn’t quite correct, but we have a good one now and it is going into 1.8.5 (just squeaked in before release) > >> On Apr 29, 2015, at 9:50 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: >> >> OpenMPI developers, >> >> We've had issues (memory errors) with OpenMPI - and code in PETSc >> library that uses MPI_Win_fence(). >> >> Vagrind shows memory corruption deep inside OpenMPI function stack. >> >> I'm attaching a potential patch that appears to fix this issue for us. >> [the corresponding valgrind trace is listed in the patch header] >> >> Perhaps there is a more appropriate fix for this memory corruption. Could >> you check on this? >> >> [Sorry I don't have a pure MPI test code to demonstrate this error - >> but a PETSc test example code consistantly reproduces this issue] >> >> Thanks, >> Satish<openmpi-1.8.4.patch>_______________________________________________ >> users mailing list >> users(a)open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: http://www.open-mpi.org/community/lists/users/2015/04/26818.php > > _______________________________________________ > users mailing list > users(a)open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: http://www.open-mpi.org/community/lists/users/2015/04/26822.php -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [petsc-dev] [OMPI users] potential bug with MPI_Win_fence() in openmpi-1.8.4
by George Bosilca 30 Apr '15

30 Apr '15

In the packed representation we store not MPI_Datatypes but a handcrafted id for each one. The 2 codes should have been in sync. I'm looking at another issue right now, and I'll come back to this one right after. Thanks for paying attention to the code. George. On Thu, Apr 30, 2015 at 3:13 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > Thanks for checking and getting a more appropriate fix in. > > I've just tried this out - and the PETSc test code runs fine with it. > > BTW: There is one inconsistancy in ompi/datatype/ompi_datatype_args.c > [that I noticed] - that you might want to check. > Perhaps the second line should be "(DC) * sizeof(MPI_Datatype)"? > > >>>>>>>>> > int length = sizeof(ompi_datatype_args_t) + (IC) * sizeof(int) + \ > (AC) * sizeof(OPAL_PTRDIFF_TYPE) + (DC) * > sizeof(MPI_Datatype); \ > > > pArgs->total_pack_size = (4 + (IC)) * sizeof(int) + \ > (AC) * sizeof(OPAL_PTRDIFF_TYPE) + (DC) * sizeof(int); \ > <<<<<<<<<<< > > Satish > > > On Thu, 30 Apr 2015, Matthew Knepley wrote: > > > On Fri, May 1, 2015 at 4:55 AM, Jeff Squyres (jsquyres) < > jsquyres(a)cisco.com> > > wrote: > > > > > Thank you! > > > > > > George reviewed your patch and adjusted it a bit. We applied it to > master > > > and it's pending to the release series (v1.8.x). > > > > > > > Was this identified by IBM? > > > > > > > https://github.com/open-mpi/ompi/commit/015d3f56cf749ee5ad9ea4428d2f5da72f9… > > > > Matt > > > > > > > Would you mind testing a nightly master snapshot? It should be in > > > tonight's build: > > > > > > http://www.open-mpi.org/nightly/master/ > > > > > > > > > > > > > On Apr 30, 2015, at 12:50 AM, Satish Balay <balay(a)mcs.anl.gov> > wrote: > > > > > > > > OpenMPI developers, > > > > > > > > We've had issues (memory errors) with OpenMPI - and code in PETSc > > > > library that uses MPI_Win_fence(). > > > > > > > > Vagrind shows memory corruption deep inside OpenMPI function stack. > > > > > > > > I'm attaching a potential patch that appears to fix this issue for > us. > > > > [the corresponding valgrind trace is listed in the patch header] > > > > > > > > Perhaps there is a more appropriate fix for this memory corruption. > Could > > > > you check on this? > > > > > > > > [Sorry I don't have a pure MPI test code to demonstrate this error - > > > > but a PETSc test example code consistantly reproduces this issue] > > > > > > > > Thanks, > > > > Satish<openmpi-1.8.4.patch> > > > > > > > > > -- > > > Jeff Squyres > > > jsquyres(a)cisco.com > > > For corporate legal information go to: > > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > > > > > > > _______________________________________________ > users mailing list > users(a)open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/04/26823.php >

1 0

Re: [petsc-dev] potential bug with MPI_Win_fence() in openmpi-1.8.4
by Jeff Squyres (jsquyres) 30 Apr '15

30 Apr '15

Oops -- that was a mistake from George when he committed the fix, and I just propagated that mistake into the v1.8 pull request. I'll fix it there, at least. But the master commit message is unfortunately going to have to stay wrong. :-( > On Apr 30, 2015, at 2:59 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > On Fri, May 1, 2015 at 4:55 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > Thank you! > > George reviewed your patch and adjusted it a bit. We applied it to master and it's pending to the release series (v1.8.x). > > Was this identified by IBM? > > https://github.com/open-mpi/ompi/commit/015d3f56cf749ee5ad9ea4428d2f5da72f9… > > Matt > > Would you mind testing a nightly master snapshot? It should be in tonight's build: > > http://www.open-mpi.org/nightly/master/ > > > > > On Apr 30, 2015, at 12:50 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > OpenMPI developers, > > > > We've had issues (memory errors) with OpenMPI - and code in PETSc > > library that uses MPI_Win_fence(). > > > > Vagrind shows memory corruption deep inside OpenMPI function stack. > > > > I'm attaching a potential patch that appears to fix this issue for us. > > [the corresponding valgrind trace is listed in the patch header] > > > > Perhaps there is a more appropriate fix for this memory corruption. Could > > you check on this? > > > > [Sorry I don't have a pure MPI test code to demonstrate this error - > > but a PETSc test example code consistantly reproduces this issue] > > > > Thanks, > > Satish<openmpi-1.8.4.patch> > > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [petsc-dev] potential bug with MPI_Win_fence() in openmpi-1.8.4
by Satish Balay 30 Apr '15

30 Apr '15

Thanks for checking and getting a more appropriate fix in. I've just tried this out - and the PETSc test code runs fine with it. BTW: There is one inconsistancy in ompi/datatype/ompi_datatype_args.c [that I noticed] - that you might want to check. Perhaps the second line should be "(DC) * sizeof(MPI_Datatype)"? >>>>>>>>> int length = sizeof(ompi_datatype_args_t) + (IC) * sizeof(int) + \ (AC) * sizeof(OPAL_PTRDIFF_TYPE) + (DC) * sizeof(MPI_Datatype); \ pArgs->total_pack_size = (4 + (IC)) * sizeof(int) + \ (AC) * sizeof(OPAL_PTRDIFF_TYPE) + (DC) * sizeof(int); \ <<<<<<<<<<< Satish On Thu, 30 Apr 2015, Matthew Knepley wrote: > On Fri, May 1, 2015 at 4:55 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> > wrote: > > > Thank you! > > > > George reviewed your patch and adjusted it a bit. We applied it to master > > and it's pending to the release series (v1.8.x). > > > > Was this identified by IBM? > > > https://github.com/open-mpi/ompi/commit/015d3f56cf749ee5ad9ea4428d2f5da72f9… > > Matt > > > > Would you mind testing a nightly master snapshot? It should be in > > tonight's build: > > > > http://www.open-mpi.org/nightly/master/ > > > > > > > > > On Apr 30, 2015, at 12:50 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > > OpenMPI developers, > > > > > > We've had issues (memory errors) with OpenMPI - and code in PETSc > > > library that uses MPI_Win_fence(). > > > > > > Vagrind shows memory corruption deep inside OpenMPI function stack. > > > > > > I'm attaching a potential patch that appears to fix this issue for us. > > > [the corresponding valgrind trace is listed in the patch header] > > > > > > Perhaps there is a more appropriate fix for this memory corruption. Could > > > you check on this? > > > > > > [Sorry I don't have a pure MPI test code to demonstrate this error - > > > but a PETSc test example code consistantly reproduces this issue] > > > > > > Thanks, > > > Satish<openmpi-1.8.4.patch> > > > > > > -- > > Jeff Squyres > > jsquyres(a)cisco.com > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > >

1 0

Re: [petsc-dev] Seeming performance regression with GAMG
by Mark Adams 30 Apr '15

30 Apr '15

> > Yes, thanks, this is the change. If I switch back to using the square > of the adjacency graph on all levels then I recover the previous > behaviour. Mark, any comments? > > The intent here is that you need to get faster coarsening on most problems, especially not high order discretizations. After the first coarse grid the number of non-zeros increases and this seems to be enough to coarsen a good rate for the problems that I test: lowish order FE elements (8 and maybe 27 node hexes). You have about 10 non-zeros per row. This is very low and squaring is not even really enough. BTW, if you can add fake zeros to make a large stencil and use a negative threshold value, GAMG will use that fake graph for coarsening, that is, it will not drop the zeros from the coarsening graph. BTW, I hated making this change for this reason but made a mistake in implementing this. I was always intending to do turn the squaring off after the first level, but realized earlier this year that I was not turning it off. Anyway, I would suggest that we turn the square graph argument into an integer for the number of levels that you want. Zero would be like false, one would be like true, and then you can add what you want. If there are no objections I will do this. It will break everyone that uses this parameter but it is a better interface. Mark > Cheers, > > Lawrence > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > > iQEcBAEBAgAGBQJVP1ipAAoJECOc1kQ8PEYve1QH/3SChuuIZhDOoOOfxYnezCuC > beXRvqrjem6SWMBIft67kvmAr9htGsiGg/vE+NRP9EwNu20kD+fYocJ+mBBcxFit > 2FdUperqA8F9gys2wJ+TijEQx58ieBNJJyPmQRa1DOmKHmD/WWcCIBV5q+yML9kt > B3z59ujUH1kJrmmpYmUk6aMUDAG0qU2KlVHX+p6zL1kV7ooGdYcKbiAbYdYrKO1l > 6vNw65+51Oiyi2X/oAsnxeJeAvMN2ljaSYLZJ1RZgAndSLi6bcZPQZvtzwXfOLs9 > C0qSPfXbd5O2DoenYnKzCS7t/kLnl0J3QimYex4tOSQxNQ7MtMG4CMNDmAIepUs= > =OY/s > -----END PGP SIGNATURE----- >

1 0

Re: [petsc-dev] potential bug with MPI_Win_fence() in openmpi-1.8.4
by Matthew Knepley 30 Apr '15

30 Apr '15

On Fri, May 1, 2015 at 4:55 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > Thank you! > > George reviewed your patch and adjusted it a bit. We applied it to master > and it's pending to the release series (v1.8.x). > Was this identified by IBM? https://github.com/open-mpi/ompi/commit/015d3f56cf749ee5ad9ea4428d2f5da72f9… Matt > Would you mind testing a nightly master snapshot? It should be in > tonight's build: > > http://www.open-mpi.org/nightly/master/ > > > > > On Apr 30, 2015, at 12:50 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > OpenMPI developers, > > > > We've had issues (memory errors) with OpenMPI - and code in PETSc > > library that uses MPI_Win_fence(). > > > > Vagrind shows memory corruption deep inside OpenMPI function stack. > > > > I'm attaching a potential patch that appears to fix this issue for us. > > [the corresponding valgrind trace is listed in the patch header] > > > > Perhaps there is a more appropriate fix for this memory corruption. Could > > you check on this? > > > > [Sorry I don't have a pure MPI test code to demonstrate this error - > > but a PETSc test example code consistantly reproduces this issue] > > > > Thanks, > > Satish<openmpi-1.8.4.patch> > > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] [OMPI users] potential bug with MPI_Win_fence() in openmpi-1.8.4
by Ralph Castain 30 Apr '15

30 Apr '15

Thanks! The patch wasn’t quite correct, but we have a good one now and it is going into 1.8.5 (just squeaked in before release) > On Apr 29, 2015, at 9:50 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > OpenMPI developers, > > We've had issues (memory errors) with OpenMPI - and code in PETSc > library that uses MPI_Win_fence(). > > Vagrind shows memory corruption deep inside OpenMPI function stack. > > I'm attaching a potential patch that appears to fix this issue for us. > [the corresponding valgrind trace is listed in the patch header] > > Perhaps there is a more appropriate fix for this memory corruption. Could > you check on this? > > [Sorry I don't have a pure MPI test code to demonstrate this error - > but a PETSc test example code consistantly reproduces this issue] > > Thanks, > Satish<openmpi-1.8.4.patch>_______________________________________________ > users mailing list > users(a)open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: http://www.open-mpi.org/community/lists/users/2015/04/26818.php

1 0

Re: [petsc-dev] potential bug with MPI_Win_fence() in openmpi-1.8.4
by Jeff Squyres (jsquyres) 30 Apr '15

30 Apr '15

Thank you! George reviewed your patch and adjusted it a bit. We applied it to master and it's pending to the release series (v1.8.x). Would you mind testing a nightly master snapshot? It should be in tonight's build: http://www.open-mpi.org/nightly/master/ > On Apr 30, 2015, at 12:50 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > OpenMPI developers, > > We've had issues (memory errors) with OpenMPI - and code in PETSc > library that uses MPI_Win_fence(). > > Vagrind shows memory corruption deep inside OpenMPI function stack. > > I'm attaching a potential patch that appears to fix this issue for us. > [the corresponding valgrind trace is listed in the patch header] > > Perhaps there is a more appropriate fix for this memory corruption. Could > you check on this? > > [Sorry I don't have a pure MPI test code to demonstrate this error - > but a PETSc test example code consistantly reproduces this issue] > > Thanks, > Satish<openmpi-1.8.4.patch> -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0