one compilation error in PETSc-dev with enabling GPU and complex number
Dear PETSc developers, With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows: for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr; PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } " When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72): here instantiation of "void thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator, Size, UnaryFunction>::operator()() [with RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51): here instantiation of "void thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction) [with NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, long, cusp::blas::detail::SCAL<std::complex<double>>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71): here instantiation of "size_t thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction, launch_by_value>::block_size_with_maximal_occupancy(size_t) [with NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136): here instantiation of "thrust::pair<size_t, size_t> thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size) [with NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145): here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51): error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" " However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex> int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x; x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0); std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha); return 0; } " When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here " There are not errors like "/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51): error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" " Furthermore, the warning information is also different between PETSc-dev and simple codes. Could you give me some suggestion for this errors? Thank you very much. Best, Yujie
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72): here instantiation of "void thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator, Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51): here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction) [with NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71): here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction, launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136): here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size) [with NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145): here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51): error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51): error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface. Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Thank you very much, Matt, You mean the headers of the simple codes, I further simply the codes as " #include <cusp/blas.h> #include <cusp/array1d.h> int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x; x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0); std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha); return 0; }" I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here " Thanks a lot. Best, Yujie On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72): here instantiation of "void thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator, Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51): here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction) [with NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71): here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction, launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136): here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size) [with NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145): here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51): error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51): error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On Sun, Jan 29, 2012 at 1:05 PM, recrusader <[email protected]> wrote:
Thank you very much, Matt,
You mean the headers of the simple codes, I further simply the codes as
This is a question for the CUSP mailing list. Thanks, Matt
" #include <cusp/blas.h> #include <cusp/array1d.h>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; }"
I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
"
Thanks a lot.
Best, Yujie
On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
here instantiation of "void thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator, Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with
ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
I have thought I send this question to CUSP mailing list. Therefore, I wish I could get the same errors using the simple codes. However, the errors disappear. Is it possible to provide simple codes with PETSc to show the errors? Thanks again. Best, Yujie On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 1:05 PM, recrusader <[email protected]> wrote:
Thank you very much, Matt,
You mean the headers of the simple codes, I further simply the codes as
This is a question for the CUSP mailing list.
Thanks,
Matt
" #include <cusp/blas.h> #include <cusp/array1d.h>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; }"
I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
"
Thanks a lot.
Best, Yujie
On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
here instantiation of "void thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator, Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with
ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On Sun, Jan 29, 2012 at 1:24 PM, recrusader <[email protected]> wrote:
I have thought I send this question to CUSP mailing list. Therefore, I wish I could get the same errors using the simple codes. However, the errors disappear.
The code below has no PETSc library calls. I really do not understand what you are asking. Can you run a simple PETSc example with the GPU? cd src/snes/examples/tutorials make ex5 ./ex5 -vec_type cusp -log_summary Matt
Is it possible to provide simple codes with PETSc to show the errors? Thanks again.
Best, Yujie
On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <[email protected]>wrote:
On Sun, Jan 29, 2012 at 1:05 PM, recrusader <[email protected]> wrote:
Thank you very much, Matt,
You mean the headers of the simple codes, I further simply the codes as
This is a question for the CUSP mailing list.
Thanks,
Matt
" #include <cusp/blas.h> #include <cusp/array1d.h>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; }"
I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
"
Thanks a lot.
Best, Yujie
On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
here instantiation of "void thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator, Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with
ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Dear Matt, Without PETSc library calls, the errors, that is "/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):>> error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE", disappear. Since I meet the errors when compiling PETSc-dev, is it possible to run PETSc example? Thank you very much. Best, Yujie On Sun, Jan 29, 2012 at 1:29 PM, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 1:24 PM, recrusader <[email protected]> wrote:
I have thought I send this question to CUSP mailing list. Therefore, I wish I could get the same errors using the simple codes. However, the errors disappear.
The code below has no PETSc library calls. I really do not understand what you are asking.
Can you run a simple PETSc example with the GPU?
cd src/snes/examples/tutorials make ex5 ./ex5 -vec_type cusp -log_summary
Matt
Is it possible to provide simple codes with PETSc to show the errors? Thanks again.
Best, Yujie
On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <[email protected]>wrote:
On Sun, Jan 29, 2012 at 1:05 PM, recrusader <[email protected]>wrote:
Thank you very much, Matt,
You mean the headers of the simple codes, I further simply the codes as
This is a question for the CUSP mailing list.
Thanks,
Matt
" #include <cusp/blas.h> #include <cusp/array1d.h>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; }"
I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
"
Thanks a lot.
Best, Yujie
On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
here instantiation of "void
thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator,
Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with
ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51):
here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67):
here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On Sun, Jan 29, 2012 at 1:35 PM, recrusader <[email protected]> wrote:
Dear Matt,
Without PETSc library calls, the errors, that is "/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):>> error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE", disappear.
Since I meet the errors when compiling PETSc-dev, is it possible to run PETSc example?
1) If you have an error compiling petsc-dev ALWAYS send make.log to [email protected] 2) petsc-dev only works with the latest cusp-dev from the repository Thanks Matt
Thank you very much.
Best, Yujie
On Sun, Jan 29, 2012 at 1:29 PM, Matthew Knepley <[email protected]>wrote:
On Sun, Jan 29, 2012 at 1:24 PM, recrusader <[email protected]> wrote:
I have thought I send this question to CUSP mailing list. Therefore, I wish I could get the same errors using the simple codes. However, the errors disappear.
The code below has no PETSc library calls. I really do not understand what you are asking.
Can you run a simple PETSc example with the GPU?
cd src/snes/examples/tutorials make ex5 ./ex5 -vec_type cusp -log_summary
Matt
Is it possible to provide simple codes with PETSc to show the errors? Thanks again.
Best, Yujie
On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <[email protected]>wrote:
On Sun, Jan 29, 2012 at 1:05 PM, recrusader <[email protected]>wrote:
Thank you very much, Matt,
You mean the headers of the simple codes, I further simply the codes as
This is a question for the CUSP mailing list.
Thanks,
Matt
" #include <cusp/blas.h> #include <cusp/array1d.h>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; }"
I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
"
Thanks a lot.
Best, Yujie
On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
here instantiation of "void
thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator,
Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with
ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51):
here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67):
here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Dear Matt, I got the help from Thrust developers. The problem is in PETSc, we define CUSPARRAY using cusp::array1d<PetscScalar, cusp::device_memory>. Generaly, PetscScalar is std::complex. However, the CUSPARRAY array cannot be decorated at GPU side. We need cusp::array1d<cusp::complex, cusp::device_memory>. Is there any simple method to process this in PETSc? Thank you very much. Best, Yujie On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 1:05 PM, recrusader <[email protected]> wrote:
Thank you very much, Matt,
You mean the headers of the simple codes, I further simply the codes as
This is a question for the CUSP mailing list.
Thanks,
Matt
" #include <cusp/blas.h> #include <cusp/array1d.h>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; }"
I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
"
Thanks a lot.
Best, Yujie
On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
here instantiation of "void thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator, Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with
ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On Tue, Feb 7, 2012 at 10:19 AM, recrusader <[email protected]> wrote:
Dear Matt,
I got the help from Thrust developers. The problem is in PETSc, we define CUSPARRAY using cusp::array1d<PetscScalar, cusp::device_memory>. Generaly, PetscScalar is std::complex. However, the CUSPARRAY array cannot be decorated at GPU side. We need cusp::array1d<cusp::complex, cusp::device_memory>. Is there any simple method to process this in PETSc?
This reply does not make any sense to me. What do you mean by the word "decorated"? If you mean that they only support the complex type cusp::complex, then I advise you to configure for real numbers. We do not support cusp::complex as our complex type. Thanks, Matt
Thank you very much.
Best, Yujie
On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <[email protected]>wrote:
On Sun, Jan 29, 2012 at 1:05 PM, recrusader <[email protected]> wrote:
Thank you very much, Matt,
You mean the headers of the simple codes, I further simply the codes as
This is a question for the CUSP mailing list.
Thanks,
Matt
" #include <cusp/blas.h> #include <cusp/array1d.h>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; }"
I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
"
Thanks a lot.
Best, Yujie
On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
here instantiation of "void thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator, Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with
ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
The following is the reply from Thrust developers, "The issue here is that the member functions of std::complex are not decorated with __host__ __device__ and therefore the compiler complains when asked to instantiate such functions in other __host__ __device__ code. If you switch std::complex to cusp::complex (which *is* decorated with __host__ __device__) then the problem should disappear." Thanks. On Tue, Feb 7, 2012 at 11:13 AM, Matthew Knepley <[email protected]> wrote:
On Tue, Feb 7, 2012 at 10:19 AM, recrusader <[email protected]> wrote:
Dear Matt,
I got the help from Thrust developers. The problem is in PETSc, we define CUSPARRAY using cusp::array1d<PetscScalar, cusp::device_memory>. Generaly, PetscScalar is std::complex. However, the CUSPARRAY array cannot be decorated at GPU side. We need cusp::array1d<cusp::complex, cusp::device_memory>. Is there any simple method to process this in PETSc?
This reply does not make any sense to me. What do you mean by the word "decorated"? If you mean that they only support the complex type cusp::complex, then I advise you to configure for real numbers. We do not support cusp::complex as our complex type.
Thanks,
Matt
Thank you very much.
Best, Yujie
On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <[email protected]>wrote:
On Sun, Jan 29, 2012 at 1:05 PM, recrusader <[email protected]>wrote:
Thank you very much, Matt,
You mean the headers of the simple codes, I further simply the codes as
This is a question for the CUSP mailing list.
Thanks,
Matt
" #include <cusp/blas.h> #include <cusp/array1d.h>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; }"
I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
"
Thanks a lot.
Best, Yujie
On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
here instantiation of "void
thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator,
Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with
ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51):
here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67):
here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On Tue, Feb 7, 2012 at 11:15 AM, recrusader <[email protected]> wrote:
The following is the reply from Thrust developers, "The issue here is that the member functions of std::complex are not decorated with __host__ __device__ and therefore the compiler complains when asked to instantiate such functions in other __host__ __device__ code. If you switch std::complex to cusp::complex (which *is* decorated with __host__ __device__) then the problem should disappear."
That makes sense, and my reply is the same. PETSc does not support that complex type. You can try to typedef PetscScalar to that type, but I have no idea what problems it might cause. Matt
Thanks.
On Tue, Feb 7, 2012 at 11:13 AM, Matthew Knepley <[email protected]>wrote:
On Tue, Feb 7, 2012 at 10:19 AM, recrusader <[email protected]> wrote:
Dear Matt,
I got the help from Thrust developers. The problem is in PETSc, we define CUSPARRAY using cusp::array1d<PetscScalar, cusp::device_memory>. Generaly, PetscScalar is std::complex. However, the CUSPARRAY array cannot be decorated at GPU side. We need cusp::array1d<cusp::complex, cusp::device_memory>. Is there any simple method to process this in PETSc?
This reply does not make any sense to me. What do you mean by the word "decorated"? If you mean that they only support the complex type cusp::complex, then I advise you to configure for real numbers. We do not support cusp::complex as our complex type.
Thanks,
Matt
Thank you very much.
Best, Yujie
On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <[email protected]>wrote:
On Sun, Jan 29, 2012 at 1:05 PM, recrusader <[email protected]>wrote:
Thank you very much, Matt,
You mean the headers of the simple codes, I further simply the codes as
This is a question for the CUSP mailing list.
Thanks,
Matt
" #include <cusp/blas.h> #include <cusp/array1d.h>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; }"
I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
"
Thanks a lot.
Best, Yujie
On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
here instantiation of "void
thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator,
Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with
ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51):
here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67):
here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Whether is it possible to find an efficient mechanism to do the conversion between std::complex and cusp::complex when the conversion is necessary. Thanks. Yujie On Tue, Feb 7, 2012 at 11:17 AM, Matthew Knepley <[email protected]> wrote:
On Tue, Feb 7, 2012 at 11:15 AM, recrusader <[email protected]> wrote:
The following is the reply from Thrust developers, "The issue here is that the member functions of std::complex are not decorated with __host__ __device__ and therefore the compiler complains when asked to instantiate such functions in other __host__ __device__ code. If you switch std::complex to cusp::complex (which *is* decorated with __host__ __device__) then the problem should disappear."
That makes sense, and my reply is the same. PETSc does not support that complex type. You can try to typedef PetscScalar to that type, but I have no idea what problems it might cause.
Matt
Thanks.
On Tue, Feb 7, 2012 at 11:13 AM, Matthew Knepley <[email protected]>wrote:
On Tue, Feb 7, 2012 at 10:19 AM, recrusader <[email protected]>wrote:
Dear Matt,
I got the help from Thrust developers. The problem is in PETSc, we define CUSPARRAY using cusp::array1d<PetscScalar, cusp::device_memory>. Generaly, PetscScalar is std::complex. However, the CUSPARRAY array cannot be decorated at GPU side. We need cusp::array1d<cusp::complex, cusp::device_memory>. Is there any simple method to process this in PETSc?
This reply does not make any sense to me. What do you mean by the word "decorated"? If you mean that they only support the complex type cusp::complex, then I advise you to configure for real numbers. We do not support cusp::complex as our complex type.
Thanks,
Matt
Thank you very much.
Best, Yujie
On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <[email protected]>wrote:
On Sun, Jan 29, 2012 at 1:05 PM, recrusader <[email protected]>wrote:
Thank you very much, Matt,
You mean the headers of the simple codes, I further simply the codes as
This is a question for the CUSP mailing list.
Thanks,
Matt
" #include <cusp/blas.h> #include <cusp/array1d.h>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; }"
I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
"
Thanks a lot.
Best, Yujie
On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
here instantiation of "void
thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator,
Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with
ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51):
here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67):
here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On Tue, Feb 7, 2012 at 11:20 AM, recrusader <[email protected]> wrote:
Whether is it possible to find an efficient mechanism to do the conversion between std::complex and cusp::complex when the conversion is necessary.
That does not matter. This is a compile error. We are not going to change this right now, and it seems like you are not going make the necessary changes, so I would say that complex numbers are not supported with our GPU code right now. The change would involve using cusp::complex for PetscScalar, and I am not sure how much work that would entail. Matt
Thanks. Yujie
On Tue, Feb 7, 2012 at 11:17 AM, Matthew Knepley <[email protected]>wrote:
On Tue, Feb 7, 2012 at 11:15 AM, recrusader <[email protected]> wrote:
The following is the reply from Thrust developers, "The issue here is that the member functions of std::complex are not decorated with __host__ __device__ and therefore the compiler complains when asked to instantiate such functions in other __host__ __device__ code. If you switch std::complex to cusp::complex (which *is* decorated with __host__ __device__) then the problem should disappear."
That makes sense, and my reply is the same. PETSc does not support that complex type. You can try to typedef PetscScalar to that type, but I have no idea what problems it might cause.
Matt
Thanks.
On Tue, Feb 7, 2012 at 11:13 AM, Matthew Knepley <[email protected]>wrote:
On Tue, Feb 7, 2012 at 10:19 AM, recrusader <[email protected]>wrote:
Dear Matt,
I got the help from Thrust developers. The problem is in PETSc, we define CUSPARRAY using cusp::array1d<PetscScalar, cusp::device_memory>. Generaly, PetscScalar is std::complex. However, the CUSPARRAY array cannot be decorated at GPU side. We need cusp::array1d<cusp::complex, cusp::device_memory>. Is there any simple method to process this in PETSc?
This reply does not make any sense to me. What do you mean by the word "decorated"? If you mean that they only support the complex type cusp::complex, then I advise you to configure for real numbers. We do not support cusp::complex as our complex type.
Thanks,
Matt
Thank you very much.
Best, Yujie
On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <[email protected]>wrote:
On Sun, Jan 29, 2012 at 1:05 PM, recrusader <[email protected]>wrote:
Thank you very much, Matt,
You mean the headers of the simple codes, I further simply the codes as
This is a question for the CUSP mailing list.
Thanks,
Matt
" #include <cusp/blas.h> #include <cusp/array1d.h>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; }"
I got the same compilation results " login1$ nvcc gputest.cu -o gputest /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46): here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
"
Thanks a lot.
Best, Yujie
On 1/29/12, Matthew Knepley <[email protected]> wrote:
On Sun, Jan 29, 2012 at 12:53 PM, recrusader <[email protected]> wrote:
Dear PETSc developers,
With your help, I can successfully PETSc-deve with enabling GPU and complex number. However, when I compiled the codes, I met some errors. I also tried to use simple codes to realize the same function. However, the errors disappear. One example is as follows:
for the function "VecScale_SeqCUSP" "#undef __FUNCT__ #define __FUNCT__ "VecScale_SeqCUSP" PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha) { CUSPARRAY *xarray; PetscErrorCode ierr;
PetscFunctionBegin; if (alpha == 0.0) { ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr); } else if (alpha != 1.0) { ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr); try { cusp::blas::scal(*xarray,alpha); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex); } ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr); } ierr = WaitForGPU();CHKERRCUSP(ierr); ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr); PetscFunctionReturn(0); } "
When I compiled PETSc-dev, I met the following errors: " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=PetscScalar]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
here instantiation of "void
thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator,
Size, UnaryFunction>::operator()() [with
RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
here instantiation of "void
thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
here instantiation of "size_t
thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, launch_by_value=true]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
here instantiation of "thrust::pair<size_t, size_t>
thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
[with
NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
here [ 6 instantiation contexts not shown ] instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::device_space_tag) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with
ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<PetscScalar, cusp::device_memory>, ScalarType=std::complex<double>]" veccusp.cu(1185): here
/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE"
" However, I further realize simiar codes as " #include <thrust/version.h> #include <cusp/version.h> #include <iostream> #include <cusp/blas.h> #include <cusp/array1d.h> #include <complex>
int main(void) { cusp::array1d<std::complex<double>, cusp::host_memory> *x;
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
std::complex<double> alpha(1,2.0); cusp::blas::scal(*x,alpha);
return 0; } "
When I complied it using "nvcc gputest.cu -o gputest", I only meet warning information as follows: " /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134): warning: calling a __host__ function from a __host__ __device__ function is not allowed detected during: instantiation of "void cusp::blas::detail::SCAL<T>::operator()(T2 &) [with T=std::complex<double>, T2=std::complex<double>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
here instantiation of "InputIterator thrust::detail::host::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
here instantiation of "InputIterator thrust::detail::dispatch::for_each(InputIterator, InputIterator, UnaryFunction, thrust::host_space_tag) [with
InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51):
here instantiation of "InputIterator thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction) [with InputIterator=thrust::detail::normal_iterator<std::complex<double> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67):
here instantiation of "void thrust::for_each(InputIterator, InputIterator, UnaryFunction) [with
InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]" (367): here instantiation of "void cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType) [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double> *>, ScalarType=std::complex<double>]" (748): here instantiation of "void cusp::blas::scal(Array &, ScalarType) [with Array=cusp::array1d<std::complex<double>, cusp::host_memory>, ScalarType=std::complex<double>]" gputest.cu(25): here
" There are not errors like
"/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
error: a value of type "int" cannot be assigned to an entity of type "_ZNSt7complexIdE9_ComplexTE" "
Furthermore, the warning information is also different between PETSc-dev and simple codes.
Could you give me some suggestion for this errors? Thank you very much.
The headers are complicated to get right. The whole point of what we did is to give a way to use GPU simply through the existing PETSc linear algebra interface.
Matt
Best, Yujie
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On Tue, Feb 7, 2012 at 11:29 AM, Matthew Knepley <[email protected]> wrote:
On Tue, Feb 7, 2012 at 11:20 AM, recrusader <[email protected]> wrote:
Whether is it possible to find an efficient mechanism to do the conversion between std::complex and cusp::complex when the conversion is necessary.
That does not matter. This is a compile error. We are not going to change this right now, and it seems like you are not going make the necessary changes, so I would say that complex numbers are not supported with our GPU code right now. The change would involve using cusp::complex for PetscScalar, and I am not sure how much work that would entail.
Matt
Matt, You might be interested to hear that the C++03 standard states that "The effect of instantiating the template complex for any type other than float, double or long double is unspecified". Thus, complex quad precision with it is probably a bad idea and, if I'm not mistaken, the standard does not state that the class must store data in the form double real, imag; so this could potentially break interfaces (e.g., to BLAS or LAPACK). Maybe it would be worthwhile to avoid usage of std::complex and simultaneously fix the compatibility issue with cusp::complex. I recently ripped std::complex out of Elemental for the above reasons. Jack
I've faced similar issues with std::complex, but the material below, which is not in the C++ standard, has suppressed my fears. Not eliminated though. http://fftw.org/doc/Complex-numbers.html (has a broken link, corrected below) http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2002/n1388.pdf Chetan From: [email protected] [mailto:[email protected]] On Behalf Of Jack Poulson Sent: Tuesday, February 07, 2012 10:10 AM To: PETSc users list Subject: Re: [petsc-users] one compilation error in PETSc-dev with enabling GPU and complex number On Tue, Feb 7, 2012 at 11:29 AM, Matthew Knepley <[email protected]> wrote: On Tue, Feb 7, 2012 at 11:20 AM, recrusader <[email protected]> wrote: Whether is it possible to find an efficient mechanism to do the conversion between std::complex and cusp::complex when the conversion is necessary. That does not matter. This is a compile error. We are not going to change this right now, and it seems like you are not going make the necessary changes, so I would say that complex numbers are not supported with our GPU code right now. The change would involve using cusp::complex for PetscScalar, and I am not sure how much work that would entail. Matt Matt, You might be interested to hear that the C++03 standard states that "The effect of instantiating the template complex for any type other than float, double or long double is unspecified". Thus, complex quad precision with it is probably a bad idea and, if I'm not mistaken, the standard does not state that the class must store data in the form double real, imag; so this could potentially break interfaces (e.g., to BLAS or LAPACK). Maybe it would be worthwhile to avoid usage of std::complex and simultaneously fix the compatibility issue with cusp::complex. I recently ripped std::complex out of Elemental for the above reasons. Jack
Thank you for the relevant links! The downside is that there does not seem to be a proposal to allow for more general base types within std::complex, e.g., __float128. This is actually my major complaint (that and fear of unstable complex arithmetic, especially division). Jack On Tue, Feb 7, 2012 at 1:19 PM, Chetan Jhurani <[email protected]>wrote:
I’ve faced similar issues with std::complex, but the material below,****
which is not in the C++ standard, has suppressed my fears. Not****
eliminated though.****
** **
http://fftw.org/doc/Complex-numbers.html (has a broken link, corrected below)****
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2002/n1388.pdf****
** **
Chetan****
** **
*From:* [email protected] [mailto: [email protected]] *On Behalf Of *Jack Poulson *Sent:* Tuesday, February 07, 2012 10:10 AM *To:* PETSc users list *Subject:* Re: [petsc-users] one compilation error in PETSc-dev with enabling GPU and complex number****
** **
On Tue, Feb 7, 2012 at 11:29 AM, Matthew Knepley <[email protected]> wrote:****
On Tue, Feb 7, 2012 at 11:20 AM, recrusader <[email protected]> wrote:* ***
Whether is it possible to find an efficient mechanism to do the conversion between std::complex and cusp::complex when the conversion is necessary.** **
** **
That does not matter. This is a compile error. We are not going to change this right now, and it seems like you are not going****
make the necessary changes, so I would say that complex numbers are not supported with our GPU code right now. The****
change would involve using cusp::complex for PetscScalar, and I am not sure how much work that would entail.****
** **
Matt****
Matt, ****
You might be interested to hear that the C++03 standard states that "The effect of instantiating the template complex for any type other than float, double or long double is unspecified". Thus, complex quad precision with it is probably a bad idea and, if I'm not mistaken, the standard does not state that the class must store data in the form
double real, imag;
so this could potentially break interfaces (e.g., to BLAS or LAPACK). Maybe it would be worthwhile to avoid usage of std::complex and simultaneously fix the compatibility issue with cusp::complex. I recently ripped std::complex out of Elemental for the above reasons.
Jack****
Jack, PetscScalar is defined in petscmath.h to be one of several things: typedef std::complex<float> PetscScalar; typedef std::complex<double> PetscScalar; typedef float complex PetscScalar; typedef double complex PetscScalar; typedef float PetscScalar; typedef double PetscScalar; typedef __float128 PetscScalar; Matt's point is that WE (the guys hacking on PETSc everyday) are not going to add other possibilities for cusp and complex at this point in time. We simply don't have the time/reason to add all this functionality now. If recursader wants that functionality he is free to hack the code and add it; if he does it in a clean way that he can provide patches then we will put the patches into petsc-dev. We simply don't have the resources to add all stuff to PETSc that anyone wants anytime and we have to focus on adding functionality that is commonly needed and will be widely used (especially within DOE). Barry On Feb 7, 2012, at 12:10 PM, Jack Poulson wrote:
On Tue, Feb 7, 2012 at 11:29 AM, Matthew Knepley <[email protected]> wrote: On Tue, Feb 7, 2012 at 11:20 AM, recrusader <[email protected]> wrote: Whether is it possible to find an efficient mechanism to do the conversion between std::complex and cusp::complex when the conversion is necessary.
That does not matter. This is a compile error. We are not going to change this right now, and it seems like you are not going make the necessary changes, so I would say that complex numbers are not supported with our GPU code right now. The change would involve using cusp::complex for PetscScalar, and I am not sure how much work that would entail.
Matt
Matt,
You might be interested to hear that the C++03 standard states that "The effect of instantiating the template complex for any type other than float, double or long double is unspecified". Thus, complex quad precision with it is probably a bad idea and, if I'm not mistaken, the standard does not state that the class must store data in the form
double real, imag;
so this could potentially break interfaces (e.g., to BLAS or LAPACK). Maybe it would be worthwhile to avoid usage of std::complex and simultaneously fix the compatibility issue with cusp::complex. I recently ripped std::complex out of Elemental for the above reasons.
Jack
Barry, Believe me, I apologize if it came across as me trying to dump something else on your plates. I was simply trying to point out some shortcomings of standard approaches to complex numbers. I know that PETSc supports quad precision for real numbers, but it seems that there isn't yet a good way to do so for the complex case, as it would require a custom complex class. If not, please let me know! Jack On Tue, Feb 7, 2012 at 1:26 PM, Barry Smith <[email protected]> wrote:
Jack,
PetscScalar is defined in petscmath.h to be one of several things:
typedef std::complex<float> PetscScalar; typedef std::complex<double> PetscScalar; typedef float complex PetscScalar; typedef double complex PetscScalar; typedef float PetscScalar; typedef double PetscScalar; typedef __float128 PetscScalar;
Matt's point is that WE (the guys hacking on PETSc everyday) are not going to add other possibilities for cusp and complex at this point in time. We simply don't have the time/reason to add all this functionality now. If recursader wants that functionality he is free to hack the code and add it; if he does it in a clean way that he can provide patches then we will put the patches into petsc-dev. We simply don't have the resources to add all stuff to PETSc that anyone wants anytime and we have to focus on adding functionality that is commonly needed and will be widely used (especially within DOE).
Barry
On Feb 7, 2012, at 12:10 PM, Jack Poulson wrote:
On Tue, Feb 7, 2012 at 11:29 AM, Matthew Knepley <[email protected]> wrote: On Tue, Feb 7, 2012 at 11:20 AM, recrusader <[email protected]> wrote: Whether is it possible to find an efficient mechanism to do the conversion between std::complex and cusp::complex when the conversion is necessary.
That does not matter. This is a compile error. We are not going to change this right now, and it seems like you are not going make the necessary changes, so I would say that complex numbers are not supported with our GPU code right now. The change would involve using cusp::complex for PetscScalar, and I am not sure how much work that would entail.
Matt
Matt,
You might be interested to hear that the C++03 standard states that "The effect of instantiating the template complex for any type other than float, double or long double is unspecified". Thus, complex quad precision with it is probably a bad idea and, if I'm not mistaken, the standard does not state that the class must store data in the form
double real, imag;
so this could potentially break interfaces (e.g., to BLAS or LAPACK). Maybe it would be worthwhile to avoid usage of std::complex and simultaneously fix the compatibility issue with cusp::complex. I recently ripped std::complex out of Elemental for the above reasons.
Jack
On Feb 7, 2012, at 2:12 PM, Jack Poulson wrote:
Barry,
Believe me, I apologize if it came across as me trying to dump something else on your plates. I was simply trying to point out some shortcomings of standard approaches to complex numbers.
No apology needed.
I know that PETSc supports quad precision for real numbers, but it seems that there isn't yet a good way to do so for the complex case, as it would require a custom complex class. If not, please let me know!
I have not tried complex quad precision. If you are right than it may not be possible right out of the box with std:complex. We are not in position to provide a proper complex class for this case (for the same reasons I listed before) so if someone wants PETSc with complex quad they may have their work cut out for them. Barry
Jack
On Tue, Feb 7, 2012 at 1:26 PM, Barry Smith <[email protected]> wrote:
Jack,
PetscScalar is defined in petscmath.h to be one of several things:
typedef std::complex<float> PetscScalar; typedef std::complex<double> PetscScalar; typedef float complex PetscScalar; typedef double complex PetscScalar; typedef float PetscScalar; typedef double PetscScalar; typedef __float128 PetscScalar;
Matt's point is that WE (the guys hacking on PETSc everyday) are not going to add other possibilities for cusp and complex at this point in time. We simply don't have the time/reason to add all this functionality now. If recursader wants that functionality he is free to hack the code and add it; if he does it in a clean way that he can provide patches then we will put the patches into petsc-dev. We simply don't have the resources to add all stuff to PETSc that anyone wants anytime and we have to focus on adding functionality that is commonly needed and will be widely used (especially within DOE).
Barry
On Feb 7, 2012, at 12:10 PM, Jack Poulson wrote:
On Tue, Feb 7, 2012 at 11:29 AM, Matthew Knepley <[email protected]> wrote: On Tue, Feb 7, 2012 at 11:20 AM, recrusader <[email protected]> wrote: Whether is it possible to find an efficient mechanism to do the conversion between std::complex and cusp::complex when the conversion is necessary.
That does not matter. This is a compile error. We are not going to change this right now, and it seems like you are not going make the necessary changes, so I would say that complex numbers are not supported with our GPU code right now. The change would involve using cusp::complex for PetscScalar, and I am not sure how much work that would entail.
Matt
Matt,
You might be interested to hear that the C++03 standard states that "The effect of instantiating the template complex for any type other than float, double or long double is unspecified". Thus, complex quad precision with it is probably a bad idea and, if I'm not mistaken, the standard does not state that the class must store data in the form
double real, imag;
so this could potentially break interfaces (e.g., to BLAS or LAPACK). Maybe it would be worthwhile to avoid usage of std::complex and simultaneously fix the compatibility issue with cusp::complex. I recently ripped std::complex out of Elemental for the above reasons.
Jack
On Tue, Feb 7, 2012 at 23:18, Barry Smith <[email protected]> wrote:
I have not tried complex quad precision. If you are right than it may not be possible right out of the box with std:complex. We are not in position to provide a proper complex class for this case (for the same reasons I listed before) so if someone wants PETSc with complex quad they may have their work cut out for them.
It's spelled __complex128. http://gcc.gnu.org/onlinedocs/gcc-4.6.2/libquadmath.pdf Perhaps it's a shame they didn't use C99 complex __float128, but such is life.
On Tue, Feb 7, 2012 at 3:34 PM, Jed Brown <[email protected]> wrote:
On Tue, Feb 7, 2012 at 23:18, Barry Smith <[email protected]> wrote:
I have not tried complex quad precision. If you are right than it may not be possible right out of the box with std:complex. We are not in position to provide a proper complex class for this case (for the same reasons I listed before) so if someone wants PETSc with complex quad they may have their work cut out for them.
It's spelled __complex128. http://gcc.gnu.org/onlinedocs/gcc-4.6.2/libquadmath.pdf
Perhaps it's a shame they didn't use C99 complex __float128, but such is life.
Good to know; I didn't know libquadmath provided that. The catch is that it does not allow one to easily extend templates that previously handled 32-bit and 64-bit base types. On the other hand, a custom complex class could easily just call the built-in functions for __complex128 when instantiated with a base type of __float128. Jack
On Wed, Feb 8, 2012 at 00:43, Jack Poulson <[email protected]> wrote:
Good to know; I didn't know libquadmath provided that. The catch is that it does not allow one to easily extend templates that previously handled 32-bit and 64-bit base types. On the other hand, a custom complex class could easily just call the built-in functions for __complex128 when instantiated with a base type of __float128.
It'll work fine with PETSc's typedef of PetscReal and PetscScalar. I never like that C++ crap anyway. ;-D (You could template over the real and complex types separately, but I guess you are assuming a certain syntax for converting between reals and complex.)
On Tue, Feb 7, 2012 at 3:46 PM, Jed Brown <[email protected]> wrote:
On Wed, Feb 8, 2012 at 00:43, Jack Poulson <[email protected]> wrote:
Good to know; I didn't know libquadmath provided that. The catch is that it does not allow one to easily extend templates that previously handled 32-bit and 64-bit base types. On the other hand, a custom complex class could easily just call the built-in functions for __complex128 when instantiated with a base type of __float128.
It'll work fine with PETSc's typedef of PetscReal and PetscScalar. I never like that C++ crap anyway. ;-D
(You could template over the real and complex types separately, but I guess you are assuming a certain syntax for converting between reals and complex.)
Yes, one could argue that PetscScalar can behave like a custom complex class, but with only a single instantiation choice through the preprocessor where each implementation was completely separate. If this type of logic was pulled into the language instead of lying at the preprocessor stage, then it would result in a custom complex class like I am arguing for. Either way std::complex is not being used for quads. Any way, it sounds like the current approach will work just fine for PETSc, so I will shut up. The only other argument would be if someone wanted to compute using some exotic datatype like the Gaussian integers, but that would require a subset of the functionality of standard complex class that only assumed a ring instead of a field. Jack
On Wed, Feb 8, 2012 at 01:12, Jack Poulson <[email protected]> wrote:
Yes, one could argue that PetscScalar can behave like a custom complex class, but with only a single instantiation choice through the preprocessor where each implementation was completely separate. If this type of logic was pulled into the language instead of lying at the preprocessor stage, then it would result in a custom complex class like I am arguing for. Either way std::complex is not being used for quads.
I think the typedefs are good enough for the 99%, but I was just teasing.
Any way, it sounds like the current approach will work just fine for PETSc, so I will shut up. The only other argument would be if someone wanted to compute using some exotic datatype like the Gaussian integers, but that would require a subset of the functionality of standard complex class that only assumed a ring instead of a field.
;-)
participants (6)
-
Barry Smith -
Chetan Jhurani -
Jack Poulson -
Jed Brown -
Matthew Knepley -
recrusader