High level GPU programming in C++

Question

I've been looking into libraries/extensions for C++ that will allow GPU-based processing on a high level. I'm not an expert in GPU programming and I don't want to dig too deep. I have a neural network consisting of classes with virtual functions. I need a library that basically does the GPU allocation for me - on a high level. There is a guy who wrote a thesis on a system called GPU++ which does most of the GPU stuff for you. I can't find the code anywhere, just his thesis.

Does anyone know of a similar library, or does anyone have the code for GPU++? Libraries like CUDA are too low level and can't handle most of my operations (at least not without rewriting all my processes and algorithms - which I don't want to do).

Solution 1

The Thrust library provides containers, parallel primitives and algorithms. All of this functionality is nicely wrapped up in a STL-like syntax. So, if you are familiar with STL, you can actually write entire CUDA programs using just Thrust, without having to write a single CUDA kernel. Have a look at the simple examples in the Quick Start Guide to see the kind of high-level programs you can write using Thrust.

Solution 2

There are many high-level libraries dedicated to GPGPU programming. Since they rely on CUDA and/or OpenCL, they have to be chosen wisely (a CUDA-based program will not run on AMD's GPUs, unless it goes through a pre-processing step with projects such as gpuocelot).

CUDA

You can find some examples of CUDA libraries on the NVIDIA website.

• Thrust: the official description speaks for itself

Thrust is a parallel algorithms library which resembles the C++ Standard Template Library (STL). Thrust's high-level interface greatly enhances programmer productivity while enabling performance portability between GPUs and multicore CPUs. Interoperability with established technologies (such as CUDA, TBB, and OpenMP) facilitates integration with existing software.

As @Ashwin pointed out, the STL-like syntax of Thrust makes it a widely chosen library when developing CUDA programs. A quick look at the examples shows the kind of code you will be writing if you decide to use this library. NVIDIA's website presents the key features of this library. A video presentation (from GTC 2012) is also available.

• CUB: the official description tells us:

CUB provides state-of-the-art, reusable software components for every layer of the CUDA programming mode. It is a flexible library of cooperative threadblock primitives and other utilities for CUDA kernel programming.

It provides device-wide, block-wide and warp-wide parallel primitives such as parallel sort, prefix scan, reduction, histogram etc.

It is open-source and available on GitHub. It is not high-level from an implementation point of view (you develop in CUDA kernels), but provides high-level algorithms and routines.

• mshadow: lightweight CPU/GPU matrix/tensor template library in C++/CUDA.

This library is mostly used for machine learning, and relies on expression templates.

• Eigen: support for CUDA with a new Tensor class have been added in version 3.3. It is used by Google in TensorFlow, and is still experimental.

Starting from Eigen 3.3, it is now possible to use Eigen's objects and algorithms within CUDA kernels. However, only a subset of features are supported to make sure that no dynamic allocation is triggered within a CUDA kernel.

OpenCL

Note that OpenCL does more than GPGPU computing, since it supports heterogeneous platforms (multi-core CPUs, GPUs etc.).

• OpenACC: this project provides OpenMP-like support for GPGPU. A large part of the programming is done implicitly by the compiler and the run-time API. You can find a sample code on their website.

The OpenACC Application Program Interface describes a collection of compiler directives to specify loops and regions of code in standard C, C++ and Fortran to be offloaded from a host CPU to an attached accelerator, providing portability across operating systems, host CPUs and accelerators.

• Bolt: open-source library with STL-like interface.

Bolt is a C++ template library optimized for heterogeneous computing. Bolt is designed to provide high-performance library implementations for common algorithms such as scan, reduce, transform, and sort. The Bolt interface was modeled on the C++ Standard Template Library (STL). Developers familiar with the STL will recognize many of the Bolt APIs and customization techniques.

• Boost.Compute: as @Kyle Lutz said, Boost.Compute provides a STL-like interface for OpenCL. Note that this is not an official Boost library (yet).

• SkelCL "is a library providing high-level abstractions for alleviated programming of modern parallel heterogeneous systems". This library relies on skeleton programming, and you can find more information in their research papers.

CUDA + OpenCL

• ArrayFire is an open-source (used to be proprietary) GPGPU programming library. They first targeted CUDA, but now support OpenCL as well. You can check the examples available online. NVIDIA's website provides a good summary of its key features.

Complementary information

Although this is not really in the scope of this question, there is also the same kind of support for other programming languages:

• Python: PyCUDA for CUDA, Clyther and PyOpenCL for OpenCL. There is a dedicated StackOverflow question for this.
• Java: JCuda for CUDA, and for OpenCL, you can check this other question.
• JavaScript: GPU.JS for WebGl.

If you need to do linear algebra (for instance) or other specific operations, dedicated math libraries are also available for CUDA and OpenCL (e.g. ViennaCL, CUBLAS, MAGMA etc.).

Also note that using these libraries does not prevent you from doing some low-level operations if you need to do some very specific computation.

Finally, we can mention the future of the C++ standard library. There has been extensive work to add parallelism support. This is still a technical specification, and GPUs are not explicitely mentioned AFAIK (although NVIDIA's Jared Hoberock, developer of Thrust, is directly involved), but the will to make this a reality is definitely there.

Solution 3

Take a look at Boost.Compute. It provides a high-level, STL-like interface including containers like vector<T> and algorithms like transform() and sort().

It's built on OpenCL allowing it to run on most modern GPUs and CPUs including those by NVIDIA, AMD, and Intel.

Solution 4

Another high level library is VexCL -- a vector expression template library for OpenCL. It provides intuitive notation for vector operations and is available under MIT license.

Solution 5

If you're looking for higher-dimensional containers and the ability to pass and manipulate these containers in kernel code, I've spent the last few years developing the ecuda API to assist in my own scientific research projects (so it's been put through the paces). Hopefully it can fill a needed niche. A brief example of how it can be used (C++11 features are used here, but ecuda will work fine with pre-C++11 compilers):

#include <cstdlib>
#include <iomanip>
#include <iostream>
#include <vector>

#include <ecuda/ecuda.hpp>

// kernel function
__global__
void calcColumnSums(
  typename ecuda::matrix<double>::const_kernel_argument mat,
  typename ecuda::vector<double>::kernel_argument vec
)
{
    const std::size_t t = threadIdx.x;
    auto col = mat.get_column(t);
    vec[t] = ecuda::accumulate( col.begin(), col.end(), static_cast<double>(0) );
}

int main( int argc, char* argv[] )
{

    // allocate 1000x1000 hardware-aligned device memory matrix
    ecuda::matrix<double> deviceMatrix( 1000, 1000 );

    // generate random values row-by-row and copy to matrix
    std::vector<double> hostRow( 1000 );
    for( std::size_t i = 0; i < 1000; ++i ) {
        for( double& x : hostRow ) x = static_cast<double>(rand())/static_cast<double>(RAND_MAX);
        ecuda::copy( hostRow.begin(), hostRow.end(), deviceMatrix[i].begin() );
    }

    // allocate device memory for column sums
    ecuda::vector<double> deviceSums( 1000 );

    CUDA_CALL_KERNEL_AND_WAIT(
        calcColumnSums<<<1,1000>>>( deviceMatrix, deviceSums )
    );

    // copy columns sums to host and print
    std::vector<double> hostSums( 1000 );
    ecuda::copy( deviceSums.begin(), deviceSums.end(), hostSums.begin() );

    std::cout << "SUMS =";
    for( const double& x : hostSums ) std::cout << " " << std::fixed << x;
    std::cout << std::endl;

    return 0;

}

I wrote it to be as an intuitive as possible (usually as simple as replacing std:: with ecuda::). If you know STL, then ecuda should do what you'd logically expect a CUDA-based C++ extension to do.

View more solutions

Comments

[stardust]

Something like this maybe? viennacl.sourceforge.net/viennacl-examples-vector.html

[ShPavel]

OpenACC openacc-standard.org or Thrust developer.nvidia.com/thrust ?

[sgarizvi]

You can try arrayfire, or OpenCV GPU Module

[Ciro Santilli OurBigBook.com]

Voting to close as tool rec.

[Anderson Green]

@CiroSantilli新疆改造中心法轮功六四事件 It's still a good question, though it's on the wrong site. Can this question be migrated to softwarerecs, or will it stay closed forever?

[Ciro Santilli OurBigBook.com]

@AndersonGreen I actually changed my philosophy since then, I now believe we should never ever close anything. It is not possible to migrate after 6 months I believe, the only options is to open a new question. softwarerecs will likely accept it.

[goocreations]

Now this is what I actually was looking for. Thank you very much.

[goocreations]

Looking for something cross-platform. AMP seems to Windows specific.

[lmat - Reinstate Monica]

@goocreations Consider marking this the correct answer ?

[The Vivandiere]

@Ashwin, there has been now a successor to Thrust called Bulk. Jared Hoberock gave a presentation on it. Do you have any opinion about it? From the presentation, it looked like highly advanced.

[mathengineer]

ecuda avoids memory transfer in following call to functions?. What is the performance?. Does it hides the complex memory allocation and transfers?

[mathengineer]

Microsoft & AMD released a C++ AMP "with linux support". Is C++AMP a higher level library that hides the Opencl or it is a different solution?

[mathengineer]

I used Eigen to invert large matrices and only runs 2x faster than using a one thread code

[Paschover]

At the time of this writing, it looks like ArrayFire is one of the few libraries still being maintained, and it seems to be the most versatile.

[paleonix]

This needs to be updated with NVIDIA MatX as well as all of the new AMD ROCM alternatives like rocThrust. With Intel GPGPUs on the Horizon, one might even include OneAPI stuff. Also there is SYCL and Kokkos trying to generally bring more portability.