November 2013 - moab-dev - mailman3.cels.anl.gov

commit/MOAB: danwu: Better logic to test whether gents has multiple sequences in ParallelComm::gather_data().
by commits-noreply＠bitbucket.org 05 Nov '13

05 Nov '13

1 new commit in MOAB: https://bitbucket.org/fathomteam/moab/commits/9fa5719c4570/ Changeset: 9fa5719c4570 Branch: master User: danwu Date: 2013-11-05 16:44:20 Summary: Better logic to test whether gents has multiple sequences in ParallelComm::gather_data(). Affected #: 1 file diff --git a/src/parallel/ParallelComm.cpp b/src/parallel/ParallelComm.cpp index efc4fa1..ab29867 100644 --- a/src/parallel/ParallelComm.cpp +++ b/src/parallel/ParallelComm.cpp @@ -8762,15 +8762,22 @@ ErrorCode ParallelComm::post_irecv(std::vector<unsigned int>& shared_procs, MPI_Gatherv(senddata, sz_buffer, MPI_BYTE, recvbuf, &recvcnts[0], &displs[0], MPI_BYTE, 0, comm()); void* gvals = NULL; - // If gents is contiguous, gathered values will be directly copied to its tag space - if (gents.psize() == 1) { + + // Test whether gents has multiple sequences + bool multiple_sequences = false; + if (gents.psize() > 1) + multiple_sequences = true; + else { int count; rval = mbImpl->tag_iterate(tag_handle, gents.begin(), gents.end(), count, gvals); assert(NULL != gvals); - assert(count == (int)gents.size()); + assert(count > 0); + if ((size_t)count != gents.size()) + multiple_sequences = true; } - // If gents is not contiguous, gathered values will be copied to a temp buffer first - else if (gents.psize() > 1) { + + // If gents has multiple sequences, create a temp buffer for gathered values + if (multiple_sequences) { gvals = new (std::nothrow) char[gents.size() * bytes_per_tag]; assert(NULL != gvals); } @@ -8785,8 +8792,8 @@ ErrorCode ParallelComm::post_irecv(std::vector<unsigned int>& shared_procs, } } - // If gents is not contiguous, set tag data (stored in the temp buffer) on each sequence separately - if (gents.psize() > 1) { + // If gents has multiple sequences, set tag data (stored in the temp buffer) on each sequence separately + if (multiple_sequences) { Range::iterator iter = gents.begin(); size_t start_idx = 0; while (iter != gents.end()) { @@ -8802,7 +8809,7 @@ ErrorCode ParallelComm::post_irecv(std::vector<unsigned int>& shared_procs, assert(start_idx == gents.size()); // Delete the temp buffer - delete (char*)gvals; + delete[] (char*)gvals; } } Repository URL: https://bitbucket.org/fathomteam/moab/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

buildbot failure in moab-occ-par-sta2: failed shell test install
by moab-dev＠mcs.anl.gov 05 Nov '13

05 Nov '13

1 0

buildbot failure in moab-occ-sersta-trunk: failed shell test install
by moab-dev＠mcs.anl.gov 05 Nov '13

05 Nov '13

1 0

buildbot failure in moab-cub-par-nopnet: failed shell test install
by moab-dev＠mcs.anl.gov 05 Nov '13

05 Nov '13

1 0

buildbot failure in moab-occ-sertru: failed shell test install
by moab-dev＠mcs.anl.gov 05 Nov '13

05 Nov '13

1 0

commit/MOAB: danwu: Thanks for Jayesh to find this bug. ParallelComm::gather_data() does not correctly handle the case when the gather set cell entities are not contiguous (e.g. MPAS gather set cells created without NO_MIXED_ELEMENTS read option). The destination of memcpy() is only for the first sequence of the cells, while the source is for all cells, causing invalid write and stack smashing. To fix it, we use a temp buffer for memcpy(), and then set tag data on each sequence of cells separatel
by commits-noreply＠bitbucket.org 04 Nov '13

04 Nov '13

1 new commit in MOAB: https://bitbucket.org/fathomteam/moab/commits/6ada315d6b44/ Changeset: 6ada315d6b44 Branch: master User: danwu Date: 2013-11-04 23:23:10 Summary: Thanks for Jayesh to find this bug. ParallelComm::gather_data() does not correctly handle the case when the gather set cell entities are not contiguous (e.g. MPAS gather set cells created without NO_MIXED_ELEMENTS read option). The destination of memcpy() is only for the first sequence of the cells, while the source is for all cells, causing invalid write and stack smashing. To fix it, we use a temp buffer for memcpy(), and then set tag data on each sequence of cells separately. Affected #: 1 file diff --git a/src/parallel/ParallelComm.cpp b/src/parallel/ParallelComm.cpp index e83653b..efc4fa1 100644 --- a/src/parallel/ParallelComm.cpp +++ b/src/parallel/ParallelComm.cpp @@ -8735,24 +8735,24 @@ ErrorCode ParallelComm::post_irecv(std::vector<unsigned int>& shared_procs, ErrorCode rval = mbImpl->tag_get_bytes(tag_handle, bytes_per_tag); if (rval != MB_SUCCESS) return rval; - int sz_buffer = sizeof(int)+gather_ents.size()*(sizeof(int)+bytes_per_tag); + int sz_buffer = sizeof(int) + gather_ents.size()*(sizeof(int) + bytes_per_tag); void* senddata = malloc(sz_buffer); ((int*)senddata)[0] = (int) gather_ents.size(); - int * ptr_int = (int*)senddata+1; + int * ptr_int = (int*)senddata + 1; rval = mbImpl->tag_get_data(id_tag, gather_ents, (void*)ptr_int); - ptr_int = (int*)(senddata)+1+gather_ents.size(); - rval = mbImpl->tag_get_data(tag_handle, gather_ents,(void*)ptr_int); + ptr_int = (int*)(senddata) + 1 + gather_ents.size(); + rval = mbImpl->tag_get_data(tag_handle, gather_ents, (void*)ptr_int); std::vector<int> displs(proc_config().proc_size(), 0); MPI_Gather(&sz_buffer, 1, MPI_INT, &displs[0], 1, MPI_INT, 0, comm()); std::vector<int> recvcnts(proc_config().proc_size(), 0); std::copy(displs.begin(), displs.end(), recvcnts.begin()); std::partial_sum(displs.begin(), displs.end(), displs.begin()); - std::vector<int>::iterator lastM1 = displs.end()-1; + std::vector<int>::iterator lastM1 = displs.end() - 1; std::copy_backward(displs.begin(), lastM1, displs.end()); //std::copy_backward(displs.begin(), --displs.end(), displs.end()); displs[0] = 0; - if (rank()!=0) + if (rank() != 0) MPI_Gatherv(senddata, sz_buffer, MPI_BYTE, NULL, NULL, NULL, MPI_BYTE, 0, comm()); else { Range gents; @@ -8761,19 +8761,51 @@ ErrorCode ParallelComm::post_irecv(std::vector<unsigned int>& shared_procs, void* recvbuf = malloc(recvbuffsz); MPI_Gatherv(senddata, sz_buffer, MPI_BYTE, recvbuf, &recvcnts[0], &displs[0], MPI_BYTE, 0, comm()); - void *gvals; - int count; - rval = mbImpl->tag_iterate(tag_handle, gents.begin(), gents.end(), count, gvals); + void* gvals = NULL; + // If gents is contiguous, gathered values will be directly copied to its tag space + if (gents.psize() == 1) { + int count; + rval = mbImpl->tag_iterate(tag_handle, gents.begin(), gents.end(), count, gvals); + assert(NULL != gvals); + assert(count == (int)gents.size()); + } + // If gents is not contiguous, gathered values will be copied to a temp buffer first + else if (gents.psize() > 1) { + gvals = new (std::nothrow) char[gents.size() * bytes_per_tag]; + assert(NULL != gvals); + } + for (int i = 0; i != (int)size(); ++i) { - int numents = *(int*)(((char*)recvbuf)+displs[i]); - int* id_ptr = (int*)(((char*)recvbuf)+displs[i]+sizeof(int)); - char* val_ptr = (char*)(id_ptr+numents); - for (int j=0; j != numents; ++j) { - int idx = id_ptr[j]; - memcpy((char*)gvals+(idx-1)*bytes_per_tag, val_ptr+j*bytes_per_tag, bytes_per_tag); - } + int numents = *(int*)(((char*)recvbuf) + displs[i]); + int* id_ptr = (int*)(((char*)recvbuf) + displs[i] + sizeof(int)); + char* val_ptr = (char*)(id_ptr + numents); + for (int j = 0; j != numents; ++j) { + int idx = id_ptr[j]; + memcpy((char*)gvals + (idx - 1)*bytes_per_tag, val_ptr + j*bytes_per_tag, bytes_per_tag); + } + } + + // If gents is not contiguous, set tag data (stored in the temp buffer) on each sequence separately + if (gents.psize() > 1) { + Range::iterator iter = gents.begin(); + size_t start_idx = 0; + while (iter != gents.end()) { + int count; + void* ptr; + rval = mbImpl->tag_iterate(tag_handle, iter, gents.end(), count, ptr); + for (int i = 0; i < count; i++) + ((char*)ptr)[i] = ((char*)gvals)[start_idx + i]; + + iter += count; + start_idx += count; + } + assert(start_idx == gents.size()); + + // Delete the temp buffer + delete (char*)gvals; } } + return MB_SUCCESS; } Repository URL: https://bitbucket.org/fathomteam/moab/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/MOAB: danwu: Optimize gather set code for MPAS reader. Use Range as the element type for gather_set_cells_with_n_edges array, instead of using vector. In create_gather_set_edges(), vertices_on_gather_set_edges can use MOAB allocated memory storage for conn_arr_gather_set_edges. In create_padded_gather_set_cells(), vertices_on_gather_set_cells can use MOAB allocated memory storage for conn_arr_gather_set_cells.
by commits-noreply＠bitbucket.org 01 Nov '13

01 Nov '13

1 new commit in MOAB: https://bitbucket.org/fathomteam/moab/commits/6aedff57f7ca/ Changeset: 6aedff57f7ca Branch: master User: danwu Date: 2013-11-01 22:26:16 Summary: Optimize gather set code for MPAS reader. Use Range as the element type for gather_set_cells_with_n_edges array, instead of using vector. In create_gather_set_edges(), vertices_on_gather_set_edges can use MOAB allocated memory storage for conn_arr_gather_set_edges. In create_padded_gather_set_cells(), vertices_on_gather_set_cells can use MOAB allocated memory storage for conn_arr_gather_set_cells. Affected #: 1 file diff --git a/src/io/NCHelperMPAS.cpp b/src/io/NCHelperMPAS.cpp index 64c8ebf..6a73e31 100644 --- a/src/io/NCHelperMPAS.cpp +++ b/src/io/NCHelperMPAS.cpp @@ -380,18 +380,17 @@ ErrorCode NCHelperMPAS::create_mesh(Range& faces) ERRORS(success, "Failed on wait_all."); #endif - // correct the connectivity array, replace the padded vertices with the last vertices in the - // corresponding elements; sometimes the padded vertices are 0, sometimes a big vertex id - // make sure they are consistent to our padded option - for (int i=0; i<nLocalCells; i++) - { - int nVertsInCell=num_edges_on_local_cells[i]; - int indxInConn=i*maxEdgesPerCell; - for (int j=nVertsInCell; j<maxEdgesPerCell; j++) - { - vertices_on_local_cells[indxInConn+j]=vertices_on_local_cells[indxInConn+nVertsInCell-1]; - } + // Correct local cell vertices array, replace the padded vertices with the last vertices + // in the corresponding cells; sometimes the padded vertices are 0, sometimes a large + // vertex id. Make sure they are consistent to our padded option + for (int local_cell_idx = 0; local_cell_idx < nLocalCells; local_cell_idx++) { + int num_edges = num_edges_on_local_cells[local_cell_idx]; + int idx_in_local_vert_arr = local_cell_idx * maxEdgesPerCell; + int last_vert_idx = vertices_on_local_cells[idx_in_local_vert_arr + num_edges - 1]; + for (int i = num_edges; i < maxEdgesPerCell; i++) + vertices_on_local_cells[idx_in_local_vert_arr + i] = last_vert_idx; } + // Create local vertices EntityHandle start_vertex; ErrorCode rval = create_local_vertices(vertices_on_local_cells, start_vertex); @@ -978,7 +977,7 @@ ErrorCode NCHelperMPAS::redistribute_local_cells(int start_cell_idx) int success = NCFUNC(inq_varid)(_fileId, "xCell", &xCellVarId); ERRORS(success, "Failed to get variable id of xCell."); std::vector<double> xCell(nLocalCells); - NCDF_SIZE read_start = static_cast<NCDF_SIZE>(start_cell_idx-1); + NCDF_SIZE read_start = static_cast<NCDF_SIZE>(start_cell_idx - 1); NCDF_SIZE read_count = static_cast<NCDF_SIZE>(nLocalCells); success = NCFUNCAG(_vara_double)(_fileId, xCellVarId, &read_start, &read_count, &xCell[0]); ERRORS(success, "Failed to read xCell data."); @@ -1188,8 +1187,7 @@ ErrorCode NCHelperMPAS::create_local_vertices(const std::vector<int>& vertices_o return MB_SUCCESS; } -ErrorCode NCHelperMPAS::create_local_edges(EntityHandle start_vertex, - const std::vector<int>& num_edges_on_local_cells) +ErrorCode NCHelperMPAS::create_local_edges(EntityHandle start_vertex, const std::vector<int>& num_edges_on_local_cells) { Interface*& mbImpl = _readNC->mbImpl; Tag& mGlobalIdTag = _readNC->mGlobalIdTag; @@ -1238,16 +1236,16 @@ ErrorCode NCHelperMPAS::create_local_edges(EntityHandle start_vertex, ERRORS(success, "Failed on wait_all."); #endif - // correct local edges in the same way as padded polygons, pad list with last edges in connectivity - for (int i=0; i<nLocalCells; i++) - { - int nEdgesInCell=num_edges_on_local_cells[i]; - int indxInConn=i*maxEdgesPerCell; - for (int j=nEdgesInCell; j<maxEdgesPerCell; j++) - { - edges_on_local_cells[indxInConn+j]=edges_on_local_cells[indxInConn+nEdgesInCell-1]; - } + // Correct local cell edges array in the same way as local cell vertices array, replace the + // padded edges with the last edges in the corresponding cells + for (int local_cell_idx = 0; local_cell_idx < nLocalCells; local_cell_idx++) { + int num_edges = num_edges_on_local_cells[local_cell_idx]; + int idx_in_local_edge_arr = local_cell_idx * maxEdgesPerCell; + int last_edge_idx = edges_on_local_cells[idx_in_local_edge_arr + num_edges - 1]; + for (int i = num_edges; i < maxEdgesPerCell; i++) + edges_on_local_cells[idx_in_local_edge_arr + i] = last_edge_idx; } + // Collect local edges std::sort(edges_on_local_cells.begin(), edges_on_local_cells.end()); std::copy(edges_on_local_cells.rbegin(), edges_on_local_cells.rend(), range_inserter(localGidEdges)); @@ -1320,8 +1318,9 @@ ErrorCode NCHelperMPAS::create_local_edges(EntityHandle start_vertex, ERRORS(success, "Failed on wait_all."); #endif - // Populate connectivity for local edges - // Convert in-place from int to EntityHandle type (reading backward is the trick) + // Populate connectivity data for local edges + // Convert in-place from int (stored in the first half) to EntityHandle + // Reading backward is the trick for (int edge_vert = nLocalEdges * 2 - 1; edge_vert >= 0; edge_vert--) { int global_vert_idx = vertices_on_local_edges[edge_vert]; // Global vertex index, 1 based int local_vert_idx = localGidVerts.index(global_vert_idx); // Local vertex index, 0 based @@ -1341,6 +1340,7 @@ ErrorCode NCHelperMPAS::create_local_cells(const std::vector<int>& vertices_on_l // Divide local cells into groups based on the number of edges Range local_cells_with_n_edges[DEFAULT_MAX_EDGES_PER_CELL + 1]; + // Insert larger values before smaller ones to increase efficiency for (int i = nLocalCells - 1; i >= 0; i--) { int num_edges = num_edges_on_local_cells[i]; local_cells_with_n_edges[num_edges].insert(localGidCells[i]); // Global cell index @@ -1381,7 +1381,7 @@ ErrorCode NCHelperMPAS::create_local_cells(const std::vector<int>& vertices_on_l // Set connectivity array with proper local vertices handles for (int j = 0; j < num_group_cells; j++) { - int global_cell_idx = (int)local_cells_with_n_edges[num_edges_per_cell][j]; // Global cell index, 1 based + EntityHandle global_cell_idx = local_cells_with_n_edges[num_edges_per_cell][j]; // Global cell index, 1 based int local_cell_idx = localGidCells.index(global_cell_idx); // Local cell index, 0 based assert(local_cell_idx != -1); @@ -1391,11 +1391,11 @@ ErrorCode NCHelperMPAS::create_local_cells(const std::vector<int>& vertices_on_l } for (int k = 0; k < num_edges_per_cell; k++) { - EntityHandle global_vert_id = vertices_on_local_cells[local_cell_idx * maxEdgesPerCell + k]; - int idx_vertex = localGidVerts.index(global_vert_id); - assert(idx_vertex != -1); + EntityHandle global_vert_idx = vertices_on_local_cells[local_cell_idx * maxEdgesPerCell + k]; // Global vertex index, 1 based + int local_vert_idx = localGidVerts.index(global_vert_idx); // Local vertex index, 0 based + assert(local_vert_idx != -1); conn_arr_local_cells_with_n_edges[num_edges_per_cell][j * num_edges_per_cell + k] = - start_vertex + idx_vertex; + start_vertex + local_vert_idx; } } } @@ -1552,40 +1552,46 @@ ErrorCode NCHelperMPAS::create_gather_set_edges(EntityHandle gather_set, EntityH // Create gather set edges EntityHandle start_edge; - EntityHandle* conn_arr_gather_edges = NULL; + EntityHandle* conn_arr_gather_set_edges = NULL; // Don't need to specify allocation number here, because we know enough edges were created before - ErrorCode rval = _readNC->readMeshIface->get_element_connect(nEdges, 2, MBEDGE, 0, start_edge, conn_arr_gather_edges); + ErrorCode rval = _readNC->readMeshIface->get_element_connect(nEdges, 2, MBEDGE, 0, start_edge, conn_arr_gather_set_edges); ERRORR(rval, "Failed to create edges."); // Add edges to the gather set - Range gather_edges_range(start_edge, start_edge + nEdges - 1); - rval = mbImpl->add_entities(gather_set, gather_edges_range); + Range gather_set_edges_range(start_edge, start_edge + nEdges - 1); + rval = mbImpl->add_entities(gather_set, gather_set_edges_range); ERRORR(rval, "Failed to add edges to the gather set."); // Read vertices on each edge int verticesOnEdgeVarId; int success = NCFUNC(inq_varid)(_fileId, "verticesOnEdge", &verticesOnEdgeVarId); ERRORS(success, "Failed to get variable id of verticesOnEdge."); - std::vector<int> vertices_on_gather_edges(nEdges * 2); + // Utilize the memory storage pointed by conn_arr_gather_set_edges + int* vertices_on_gather_set_edges = (int*) conn_arr_gather_set_edges; NCDF_SIZE read_starts[2] = {0, 0}; NCDF_SIZE read_counts[2] = {static_cast<NCDF_SIZE>(nEdges), 2}; #ifdef PNETCDF_FILE // Enter independent I/O mode, since this read is only for the gather processor success = NCFUNC(begin_indep_data)(_fileId); ERRORS(success, "Failed to begin independent I/O mode."); - success = NCFUNCG(_vara_int)(_fileId, verticesOnEdgeVarId, read_starts, read_counts, &vertices_on_gather_edges[0]); + success = NCFUNCG(_vara_int)(_fileId, verticesOnEdgeVarId, read_starts, read_counts, vertices_on_gather_set_edges); ERRORS(success, "Failed to read verticesOnEdge data."); success = NCFUNC(end_indep_data)(_fileId); ERRORS(success, "Failed to end independent I/O mode."); #else - success = NCFUNCG(_vara_int)(_fileId, verticesOnEdgeVarId, read_starts, read_counts, &vertices_on_gather_edges[0]); + success = NCFUNCG(_vara_int)(_fileId, verticesOnEdgeVarId, read_starts, read_counts, vertices_on_gather_set_edges); ERRORS(success, "Failed to read verticesOnEdge data."); #endif - std::copy(vertices_on_gather_edges.begin(), vertices_on_gather_edges.end(), conn_arr_gather_edges); - for (int i = 0; i < 2 * nEdges; i++) + // Populate connectivity data for gather set edges + // Convert in-place from int (stored in the first half) to EntityHandle + // Reading backward is the trick + for (int edge_vert = nEdges * 2 - 1; edge_vert >= 0; edge_vert--) { + int gather_set_vert_idx = vertices_on_gather_set_edges[edge_vert]; // Global vertex index, 1 based + gather_set_vert_idx--; // 1 based -> 0 based // Connectivity array is shifted by where the gather set vertices start - conn_arr_gather_edges[i] += gather_set_start_vertex - 1; + conn_arr_gather_set_edges[edge_vert] = gather_set_start_vertex + gather_set_vert_idx; + } return MB_SUCCESS; } @@ -1598,19 +1604,19 @@ ErrorCode NCHelperMPAS::create_gather_set_cells(EntityHandle gather_set, EntityH int nEdgesOnCellVarId; int success = NCFUNC(inq_varid)(_fileId, "nEdgesOnCell", &nEdgesOnCellVarId); ERRORS(success, "Failed to get variable id of nEdgesOnCell."); - std::vector<int> num_edges_on_gather_cells(nCells); + std::vector<int> num_edges_on_gather_set_cells(nCells); NCDF_SIZE read_start = 0; NCDF_SIZE read_count = static_cast<NCDF_SIZE>(nCells); #ifdef PNETCDF_FILE // Enter independent I/O mode, since this read is only for the gather processor success = NCFUNC(begin_indep_data)(_fileId); ERRORS(success, "Failed to begin independent I/O mode."); - success = NCFUNCG(_vara_int)(_fileId, nEdgesOnCellVarId, &read_start, &read_count, &num_edges_on_gather_cells[0]); + success = NCFUNCG(_vara_int)(_fileId, nEdgesOnCellVarId, &read_start, &read_count, &num_edges_on_gather_set_cells[0]); ERRORS(success, "Failed to read nEdgesOnCell data."); success = NCFUNC(end_indep_data)(_fileId); ERRORS(success, "Failed to end independent I/O mode."); #else - success = NCFUNCG(_vara_int)(_fileId, nEdgesOnCellVarId, &read_start, &read_count, &num_edges_on_gather_cells[0]); + success = NCFUNCG(_vara_int)(_fileId, nEdgesOnCellVarId, &read_start, &read_count, &num_edges_on_gather_set_cells[0]); ERRORS(success, "Failed to read nEdgesOnCell data."); #endif @@ -1618,52 +1624,57 @@ ErrorCode NCHelperMPAS::create_gather_set_cells(EntityHandle gather_set, EntityH int verticesOnCellVarId; success = NCFUNC(inq_varid)(_fileId, "verticesOnCell", &verticesOnCellVarId); ERRORS(success, "Failed to get variable id of verticesOnCell."); - std::vector<int> vertices_on_gather_cells(nCells * maxEdgesPerCell); + std::vector<int> vertices_on_gather_set_cells(nCells * maxEdgesPerCell); NCDF_SIZE read_starts[2] = {0, 0}; - NCDF_SIZE read_counts[2] = {static_cast<NCDF_SIZE>(nCells), maxEdgesPerCell}; + NCDF_SIZE read_counts[2] = {static_cast<NCDF_SIZE>(nCells), static_cast<NCDF_SIZE>(maxEdgesPerCell)}; #ifdef PNETCDF_FILE // Enter independent I/O mode, since this read is only for the gather processor success = NCFUNC(begin_indep_data)(_fileId); ERRORS(success, "Failed to begin independent I/O mode."); - success = NCFUNCG(_vara_int)(_fileId, verticesOnCellVarId, read_starts, read_counts, &vertices_on_gather_cells[0]); + success = NCFUNCG(_vara_int)(_fileId, verticesOnCellVarId, read_starts, read_counts, &vertices_on_gather_set_cells[0]); ERRORS(success, "Failed to read verticesOnCell data."); success = NCFUNC(end_indep_data)(_fileId); ERRORS(success, "Failed to end independent I/O mode."); #else - success = NCFUNCG(_vara_int)(_fileId, verticesOnCellVarId, read_starts, read_counts, &vertices_on_gather_cells[0]); + success = NCFUNCG(_vara_int)(_fileId, verticesOnCellVarId, read_starts, read_counts, &vertices_on_gather_set_cells[0]); ERRORS(success, "Failed to read verticesOnCell data."); #endif // Divide gather set cells into groups based on the number of edges - std::vector<int> gather_cells_with_n_edges[DEFAULT_MAX_EDGES_PER_CELL + 1]; - for (int i = 0; i < nCells; i++) { - int num_edges = num_edges_on_gather_cells[i]; - gather_cells_with_n_edges[num_edges].push_back(i + 1); // 0 based -> 1 based + Range gather_set_cells_with_n_edges[DEFAULT_MAX_EDGES_PER_CELL + 1]; + // Insert larger values before smaller values to increase efficiency + for (int i = nCells - 1; i >= 0; i--) { + int num_edges = num_edges_on_gather_set_cells[i]; + gather_set_cells_with_n_edges[num_edges].insert(i + 1); // 0 based -> 1 based } // Create gather set cells - EntityHandle* conn_arr_gather_cells_with_n_edges[DEFAULT_MAX_EDGES_PER_CELL + 1]; + EntityHandle* conn_arr_gather_set_cells_with_n_edges[DEFAULT_MAX_EDGES_PER_CELL + 1]; for (int num_edges_per_cell = 3; num_edges_per_cell <= maxEdgesPerCell; num_edges_per_cell++) { - int num_group_cells = gather_cells_with_n_edges[num_edges_per_cell].size(); + int num_group_cells = gather_set_cells_with_n_edges[num_edges_per_cell].size(); if (num_group_cells > 0) { EntityHandle start_element; ErrorCode rval = _readNC->readMeshIface->get_element_connect(num_group_cells, num_edges_per_cell, MBPOLYGON, 0, start_element, - conn_arr_gather_cells_with_n_edges[num_edges_per_cell], num_group_cells); + conn_arr_gather_set_cells_with_n_edges[num_edges_per_cell], num_group_cells); ERRORR(rval, "Failed to create cells."); // Add cells to the gather set - Range gather_cells_range(start_element, start_element + num_group_cells - 1); - rval = mbImpl->add_entities(gather_set, gather_cells_range); + Range gather_set_cells_range(start_element, start_element + num_group_cells - 1); + rval = mbImpl->add_entities(gather_set, gather_set_cells_range); ERRORR(rval, "Failed to add cells to the gather set."); for (int j = 0; j < num_group_cells; j++) { - int gather_cell_idx = gather_cells_with_n_edges[num_edges_per_cell][j]; // Global cell index, 1 based - gather_cell_idx--; // 1 based -> 0 based + int gather_set_cell_idx = gather_set_cells_with_n_edges[num_edges_per_cell][j]; // Global cell index, 1 based + gather_set_cell_idx--; // 1 based -> 0 based + + for (int k = 0; k < num_edges_per_cell; k++) { + EntityHandle gather_set_vert_idx = vertices_on_gather_set_cells[gather_set_cell_idx * maxEdgesPerCell + k]; // Global vertex index, 1 based + gather_set_vert_idx--; // 1 based -> 0 based - for (int k = 0; k < num_edges_per_cell; k++) // Connectivity array is shifted by where the gather set vertices start - conn_arr_gather_cells_with_n_edges[num_edges_per_cell][j * num_edges_per_cell + k] = - (gather_set_start_vertex - 1) + vertices_on_gather_cells[gather_cell_idx * maxEdgesPerCell + k]; + conn_arr_gather_set_cells_with_n_edges[num_edges_per_cell][j * num_edges_per_cell + k] = + gather_set_start_vertex + gather_set_vert_idx; + } } } } @@ -1679,64 +1690,73 @@ ErrorCode NCHelperMPAS::create_padded_gather_set_cells(EntityHandle gather_set, int nEdgesOnCellVarId; int success = NCFUNC(inq_varid)(_fileId, "nEdgesOnCell", &nEdgesOnCellVarId); ERRORS(success, "Failed to get variable id of nEdgesOnCell."); - std::vector<int> num_edges_on_gather_cells(nCells); + std::vector<int> num_edges_on_gather_set_cells(nCells); NCDF_SIZE read_start = 0; NCDF_SIZE read_count = static_cast<NCDF_SIZE>(nCells); #ifdef PNETCDF_FILE // Enter independent I/O mode, since this read is only for the gather processor success = NCFUNC(begin_indep_data)(_fileId); ERRORS(success, "Failed to begin independent I/O mode."); - success = NCFUNCG(_vara_int)(_fileId, nEdgesOnCellVarId, &read_start, &read_count, &num_edges_on_gather_cells[0]); + success = NCFUNCG(_vara_int)(_fileId, nEdgesOnCellVarId, &read_start, &read_count, &num_edges_on_gather_set_cells[0]); ERRORS(success, "Failed to read nEdgesOnCell data."); success = NCFUNC(end_indep_data)(_fileId); ERRORS(success, "Failed to end independent I/O mode."); #else - success = NCFUNCG(_vara_int)(_fileId, nEdgesOnCellVarId, &read_start, &read_count, &num_edges_on_gather_cells[0]); + success = NCFUNCG(_vara_int)(_fileId, nEdgesOnCellVarId, &read_start, &read_count, &num_edges_on_gather_set_cells[0]); ERRORS(success, "Failed to read nEdgesOnCell data."); #endif + // Create gather set cells + EntityHandle start_element; + EntityHandle* conn_arr_gather_set_cells = NULL; + // Don't need to specify allocation number here, because we know enough cells were created before + ErrorCode rval = _readNC->readMeshIface->get_element_connect(nCells, maxEdgesPerCell, MBPOLYGON, 0, start_element, conn_arr_gather_set_cells); + ERRORR(rval, "Failed to create cells."); + + // Add cells to the gather set + Range gather_set_cells_range(start_element, start_element + nCells - 1); + rval = mbImpl->add_entities(gather_set, gather_set_cells_range); + ERRORR(rval, "Failed to add cells to the gather set."); + // Read vertices on each gather set cell (connectivity) int verticesOnCellVarId; success = NCFUNC(inq_varid)(_fileId, "verticesOnCell", &verticesOnCellVarId); ERRORS(success, "Failed to get variable id of verticesOnCell."); - std::vector<int> vertices_on_gather_cells(nCells * maxEdgesPerCell); + // Utilize the memory storage pointed by conn_arr_gather_set_cells + int* vertices_on_gather_set_cells = (int*) conn_arr_gather_set_cells; NCDF_SIZE read_starts[2] = {0, 0}; - NCDF_SIZE read_counts[2] = {static_cast<NCDF_SIZE>(nCells), maxEdgesPerCell}; + NCDF_SIZE read_counts[2] = {static_cast<NCDF_SIZE>(nCells), static_cast<NCDF_SIZE>(maxEdgesPerCell)}; #ifdef PNETCDF_FILE // Enter independent I/O mode, since this read is only for the gather processor success = NCFUNC(begin_indep_data)(_fileId); ERRORS(success, "Failed to begin independent I/O mode."); - success = NCFUNCG(_vara_int)(_fileId, verticesOnCellVarId, read_starts, read_counts, &vertices_on_gather_cells[0]); + success = NCFUNCG(_vara_int)(_fileId, verticesOnCellVarId, read_starts, read_counts, vertices_on_gather_set_cells); ERRORS(success, "Failed to read verticesOnCell data."); success = NCFUNC(end_indep_data)(_fileId); ERRORS(success, "Failed to end independent I/O mode."); #else - success = NCFUNCG(_vara_int)(_fileId, verticesOnCellVarId, read_starts, read_counts, &vertices_on_gather_cells[0]); + success = NCFUNCG(_vara_int)(_fileId, verticesOnCellVarId, read_starts, read_counts, vertices_on_gather_set_cells); ERRORS(success, "Failed to read verticesOnCell data."); #endif - // Create gather set cells - EntityHandle start_element; - EntityHandle* conn_arr_gather_cells = NULL; - // Don't need to specify allocation number here, because we know enough cells were created before - ErrorCode rval = _readNC->readMeshIface->get_element_connect(nCells, maxEdgesPerCell, MBPOLYGON, 0, start_element, conn_arr_gather_cells); - ERRORR(rval, "Failed to create cells."); - - // Add cells to the gather set - Range gather_cells_range(start_element, start_element + nCells - 1); - rval = mbImpl->add_entities(gather_set, gather_cells_range); - ERRORR(rval, "Failed to add cells to the gather set."); - - for (int gather_cell_idx = 0; gather_cell_idx < nCells; gather_cell_idx++) { - int num_edges = num_edges_on_gather_cells[gather_cell_idx]; - for (int i = 0; i < num_edges; i++) - // Connectivity array is shifted by where the gather set vertices start - conn_arr_gather_cells[gather_cell_idx * maxEdgesPerCell + i] = (gather_set_start_vertex - 1) + vertices_on_gather_cells[gather_cell_idx * maxEdgesPerCell + i]; - - // Padding: fill connectivity array with last vertex handle - EntityHandle last_vert_handle = conn_arr_gather_cells[gather_cell_idx * maxEdgesPerCell + num_edges - 1]; + // Correct gather set cell vertices array in the same way as local cell vertices array, + // replace the padded vertices with the last vertices in the corresponding cells + for (int gather_set_cell_idx = 0; gather_set_cell_idx < nCells; gather_set_cell_idx++) { + int num_edges = num_edges_on_gather_set_cells[gather_set_cell_idx]; + int idx_in_gather_set_vert_arr = gather_set_cell_idx * maxEdgesPerCell; + int last_vert_idx = vertices_on_gather_set_cells[idx_in_gather_set_vert_arr + num_edges - 1]; for (int i = num_edges; i < maxEdgesPerCell; i++) - conn_arr_gather_cells[gather_cell_idx * maxEdgesPerCell + i] = last_vert_handle; + vertices_on_gather_set_cells[idx_in_gather_set_vert_arr + i] = last_vert_idx; + } + + // Populate connectivity data for gather set cells + // Convert in-place from int (stored in the first half) to EntityHandle + // Reading backward is the trick + for (int cell_vert = nCells * maxEdgesPerCell - 1; cell_vert >= 0; cell_vert--) { + int gather_set_vert_idx = vertices_on_gather_set_cells[cell_vert]; // Global vertex index, 1 based + gather_set_vert_idx--; // 1 based -> 0 based + // Connectivity array is shifted by where the gather set vertices start + conn_arr_gather_set_cells[cell_vert] = gather_set_start_vertex + gather_set_vert_idx; } return MB_SUCCESS; Repository URL: https://bitbucket.org/fathomteam/moab/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0