Hi, I have been thinking about the parallelization of the mesh loading recently. I guess the easiest and dirtiest way to implement is to do the simulation in two steps: 1. Run the code on a large shared memory system (with more than 64GB RAM), only to distribute the mesh to the desired number of processes and then save the sieve information of each process into files and then simply stop. This is possible since it avoids the large amount of memory usage in a real calculation such as matrices and vectors. 2. With the sieve information now stored in files, run the same sized full calculation on a production system (could be a distributed memory cluster), where instead of reading the whole mesh on the root process and then distributing it, we directly read the sieve information from the files and construct the local mesh out of it. If this is done correctly, it should end up with exactly the same distributed mesh as in step 1. Then matrices and vectors can be created and production simulations will be carried out. This method sounds very dirty but it avoids dealing with the detailed mesh distribution and thus may be easier for me to implement. I could call the view() function on each sieve to dump its information to a file. What I am using is
const Obj<ALE::Mesh::sieve_type>&sieve= m->getSieve(); //sieve->view("mesh sieve",comm);
ostringstream filename; std::ofstream fs; filename<<basename<<"_sieveNP"<<size<<"Rank"<<rank<<".txt"; fs.open(filename.str().c_str());
double rawdata=true; sieve->view(fs,filename.str().c_str(),rawdata);
fs.close(); MPI_Barrier(comm);
At this stage, I am not very sure whether the generated file will provide sufficient data to reconstruct the same sieve. Hopefully they do. I am trying to work out a way to construct sieves from these dumped files and then construct the local mesh. I will report back on the progress. At this stage, I would love to hear some insider comment on this approach. It is definitely not the best way but I hope it is feasible since it seems the easiest for me. Thank you very much. Shi On Sun, Mar 9, 2008 at 12:18 AM, Shi Jin <[email protected]> wrote:
Hi,
Thanks to your help, my CFD code using sieve is close to complete. One last part to do is to enable each process to load its own grid in parallel. Currently, I first ask the master process to load the whole mesh and call MeshDistribute() to distribute the smaller local mesh into each process. This method works fine if the mesh is small to medium in size. As the mesh gets very large, the memory require to store the whole mesh on a single process becomes impossible (this way, the memory of the master process is much larger than the others). Since each process only needs its own portion of the mesh anyway, it should just load its own. I am wondering if some help on this topic can be provided from the sieve-dev team. Thanks a lot.
I have thought about a few possibilities: 1. Like the example codes, I can use tetgen to generate the mesh on the fly and then refine it. However, this way the domain decomposition is quite random which does not work with our particle simulation code where a clearly defined domain boundary (such as some z values) is necessary to track the particles. 2. I can load a coarse mesh that I am using right now. After calling MeshDistribute() to obtain the correct domain decomposition, I can then refine each mesh using tetgen to the required resolution. However, when I tried this method, the refined mesh is actually repeating the coarse elements many times (thus although the number of nodes and vertexes are larger, the mesh is the same as before). 3. Still based on the current code, each process will try to load the same input global mesh file. Instead of loading everything, each process will only take the data that is within the pre-defined domain and throw away the outside data. This way, I can completely separate the mesh generator and the CFD code (the mesh can be generated with some sophisticated software separately). But it is important to keep the data structure across process consistent, i.e., the numbering and the ghost points must make sense. I think this is exactly what the MeshDistribute() code does and it is the perfect solution. However, to do this, one may need a deeper understanding of the sieve library.
I think the third possibility is the best and if this is not already implemented, it is probably good to have some kind of function like it in the library. I am more than happy to contribute my code to this development and of course I need some guidance. Thank you very much.
Shi
-- Sincerely, Shi Jin, Ph.D. http://www.ualberta.ca/~sjin1/ <http://www.ualberta.ca/%7Esjin1/>
-- Sincerely, Shi Jin, Ph.D. http://www.ualberta.ca/~sjin1/