Category: NDK

GPGPU In Android: Load a texture with floats and read it back

[Download code for the project: GitHub]

This post has a different flavor when compared to previous tutorials. It talks about converting the GPU in your mobile device into a powerful computing device. This is called general-purpose computing on graphics processing units or GPGPU. GPGPU a broad area of research and its applications are growing along with the computational power of the GPU.

GPGPU in computing literature mostly refers to using one of these APIs: OpenCL, CUDA, DirectCompute, compute shaders in OpenGL. Among these, only compute shaders are available in Android devices that support OpenGL ES 3.1. As is evident from Android’s dashboard, that’s a small percentage of devices (11% when this post was written). Hence we will choose the next best option that offers more support for GPGPU; OpenGL ES 3.0. In a series of posts, we will show how to write shaders in GLSL 3.0 that can perform computations on the mobile GPU.

So far we have got used to thinking of textures in GLES as images. It’s time to treat textures with more respect. We had mentioned in an earlier post that textures are viewed in GLES as 2D arrays. This is an important fact and you should allow it to completely sink in 🙂 In GPGPU we will extensively use textures as input arrays to mathematical computations as well as output arrays that store the results.

Many posts on GPGPU on the web are only concerned with using the GPU for image processing tasks. In those cases, the inputs and outputs are images or RGBA arrays stored as unsigned ints. This is a restrictive use of the GPU’s computational capabilities. In fact most applications of GPGPU listed here require the GPU to perform complex operations involving floating point numbers. So let us start by understanding how to use a GLES texture as an array of floats. We begin by taking a small step in this post:

  • Load a GLES texture with floating point values from a OpenCV Mat and read it back.

This series of tutorials on GPGPU is heavily inspired by Dominik Göddeke’s excellent post on the same topic.

Get the code

This project is available on GitHub. You can find more instructions to compile the project here.

This project requires devices that support OpenGL ES 3+.

Code overview

We retain the project structure from the previous tutorial, though most of the files are redundant for this project. We have deleted the assets folder and Assimp’s libraries to reduce size of the apk. Let us look at the new and modified files (all paths are relative to <project_path>/app/src/main):
  • com.anandmuralidhar.floattextureandroid/FloatTextureActivity.java: As always we have only one activity in the project that is defined in this file. Its JNI calls are implemented in jni/jniCalls/floatTextureActivity.cpp.
  • jni/nativeCode/floatTextureClass/floatTextureClass.cpp: It has methods of FloatTextureClass that call functions to initialize, load, and read a texture of floats.
  • jni/nativeCode/common/texture.cpp: This file is the main addition in this project. It implements all functions related to handling textures in GLES that are called by FloatTextureClass.
  • jni/nativeCode/common/myGLFunctions.cpp: We have modified the function MyGLInits so that it returns the GLES version.
  • AndroidManifest.xml: We indicate the minimum GLES version required for the app as 3.

Textures as arrays

At the sake of sounding repetitive: textures in GLES are simply arrays. We will focus on 2D arrays in this post since we have dealt with 2D textures in earlier posts. If we want to render to a texture then we need to indicate to GLES that we are not interested in rendering to the display. If you look at this project’s structure, then it is very similar to the first tutorial that simply created a GLES display. In fact, in this project we go through the same procedure as in the first tutorial so that we can get a GLES context.

Look at FloatTextureClass::PerformGLInits. First we check if GLES 3+ is supported in the device: 

glesVersion = MyGLInits();

if(glesVersion != 3) {
    // cannot proceed further since GLES 3 is not available
    initsDone = true;
    return;
}

As mentioned before, GLES 3 is a prerequisite for this project and reasons for this will be clear soon. We have defined an AsyncTask called AsyncCheckGLESVersion in FloatTextureActivity that checks glesVersion and exits the app if GLES 3+ is not supported. We have discussed AyncTask in an earlier tutorial and will skip details of AsyncCheckGLESVersion.

Next we indicate to GLES that it should not render to the display. This is done by creating an offscreen buffer as the rendering target. Such an offscreen buffer is called a framebuffer object or FBO in OpenGL ES: 

// create FBO for offscreen rendering
GLuint fb;
glGenFramebuffers(1, &fb);
// bind offscreen framebuffer (that is, skip the window-specific render target)
glBindFramebuffer(GL_FRAMEBUFFER, fb);

glGenFramebuffers will create the FBO and glBindFramebuffer will bind it. Note that the default FBO corresponding to the GLES display has fb = 0. You may wonder why we are choosing a FBO as a rendering target when we do not plan to carry out any rendering in this project. It turns out that even to read the contents of a texture, we need to create a FBO and bind to it.

Then we call InitLoadReadTexture that in turn calls functions from texture.cpp to initialize, load, and read contents of a texture. We will look at all these operations in detail soon. At the end of PerformGLInits, we bind the default FBO: 

glBindFramebuffer(GL_FRAMEBUFFER, 0);

Note that FloatTextureClass::Render does nothing and will simply display a white screen.

Now let’s see how to manipulate float textures. We will look at the following:

  1. Initialize a 2D texture to contain floating point values.
  2. Load the texture with contents of a OpenCV Mat.
  3. Read the contents of the texture to verify its values.
  4. Set a texture as the output of a rendering call, though we will perform the actual rendering in a subsequent tutorial.

Initialize a 2D float texture

Let us look at the function InitializeFloatTexture in texture.cpp. In the beginning, depending on the number of channels in the texture we set parameters that will be passed to GLES: 

GLenum internalFormat, textureFormat;
if (numberOfChannels == 1) {
    internalFormat = GL_R32F;
    textureFormat = GL_RED;
} else if (numberOfChannels == 2) {
    internalFormat = GL_RG32F;
    textureFormat = GL_RG;
} else if (numberOfChannels == 3) {
    internalFormat = GL_RGB32F;
    textureFormat = GL_RGB;
} else if (numberOfChannels == 4) {
    internalFormat = GL_RGBA32F;
    textureFormat = GL_RGBA;
}

What does channels in a texture refer to? We are only concerned with 2D textures that effectively represent 2D arrays. Now imagine a 2D array that stores an image. An image has 3 channels corresponding to the R, G, and B values. So the corresponding texture that stores this image will store each channel as a 2D array. This concept can be extended to floating point textures where each channel corresponds to a 2D array of floats. In the above code we indicate that the texture will store 32 bit floating point values.

Rest of the function is fairly similar to what we had seen earlier: 

// generate it
glGenTextures(1, &textureName);

// make active and bind
glBindTexture(GL_TEXTURE_2D, textureName);

// turn off filtering and wrap modes
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

// define texture with float format
glTexImage2D(GL_TEXTURE_2D, 0, internalFormat, textureWidth,
             textureHeight, 0, textureFormat, GL_FLOAT, 0);

glBindTexture(GL_TEXTURE_2D, 0);

Note that we do not choose any filtering modes and specify to GLES that we want to create a floating point texture by passing GL_FLOAT as texture format to glTexImage2D.

Load texture with OpenCV Mat

Look at the function LoadFloatTexture in texture.cpp. This function is fairly easy to understand since it uses concepts that are known to us from before. Briefly, we set the texture format depending on the number of channels: 

GLenum textureFormat;
if (numberOfChannels == 1) {
    textureFormat = GL_RED;
} else if (numberOfChannels == 2) {
    textureFormat = GL_RG;
} else if (numberOfChannels == 3) {
    textureFormat = GL_RGB;
} else if (numberOfChannels == 4) {
    textureFormat = GL_RGBA;
}

Since the texture is already created and memory has been provisioned for it, we upload the contents of the OpenCV Mat to it: 

glBindTexture(GL_TEXTURE_2D, textureName);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, textureWidth, textureHeight, textureFormat, GL_FLOAT,
                inputData.data);

We assume that the OpenCV Mat inputData has the same dimensions as the texture, else we can potentially cause a crash in the app.

Read contents of texture

Look at the function ReadFloatTexture in texture.cpp. In order to read the contents of a texture, we attach the texture to the FBO: 

glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
                       GL_TEXTURE_2D, textureName, 0);

If you look at the arguments to glFramebufferTexture2D in its documentation, the first argument is always GL_FRAMEBUFFER. Second argument is one the point to which the texture is attached. Third and fourth arguments are fairly obvious based on the texture’s properties and name. The last argument should always be 0.

Then we call CheckFramebufferStatus that in turn calls glCheckFramebufferStatus to check for framebuffer completeness. It is important to call this function to ensure that there are no errors in setting up the framebuffer. Common errors are not initializing the FBO, not generating the texture, or generating a texture whose format is not supported for attaching to the FBO.

Next we indicate to GLES that we plan to read the contents from the FBO attachment point GL_COLOR_ATTACHMENT0: 

glReadBuffer(GL_COLOR_ATTACHMENT0);

Then we choose the texture formats and create an empty OpenCV Mat into which we can read the texture contents: 

GLenum textureFormat;
int matType;
if (numberOfChannels == 1) {
    textureFormat = GL_RED;
    matType = CV_32F;
} else if (numberOfChannels == 2) {
    textureFormat = GL_RG;
    matType = CV_32FC2;
} else if (numberOfChannels == 3) {
    textureFormat = GL_RGB;
    matType = CV_32FC3;
} else if (numberOfChannels == 4) {
    textureFormat = GL_RGBA;
    matType = CV_32FC4;
}
cv::Mat outputMat = cv::Mat::zeros(textureHeight, textureWidth, matType);

Finally we read the texture values with glReadPixels: 

glReadPixels(0, 0, textureWidth, textureHeight, textureFormat,
             GL_FLOAT, outputMat.data);

Putting things together: initialize float texture, load with random values, and read texture contents

Let us see how to use various functions that we discussed in the above sections. Look at the function InitLoadReadTexture in floatTexture.cpp. This function is called in FloatTextureClass::PerformGLInits after the FBO is created and bound for offscreen rendering. In InitLoadReadTexture, we choose the dimensions and number of channels in the texture: 

int textureWidth = 512;
int textureHeight = 512;
int numberOfChannels = 4;

// initialize a float texture
GLuint textureName;
InitializeFloatTexture(textureName, textureWidth, textureHeight, numberOfChannels);

Then we create a OpenCV Mat with random floating point entries and load the matrix into the texture: 

// create a matrix with random entries
int matType;
if (numberOfChannels == 1) {
    matType = CV_32F;
} else if (numberOfChannels == 2) {
    matType = CV_32FC2;
} else if (numberOfChannels == 3) {
    matType = CV_32FC3;
} else if (numberOfChannels == 4) {
    matType = CV_32FC4;
}
cv::Mat randomMat(textureHeight, textureWidth, matType);
cv::randu(randomMat, -200, 200); // entries of the mat are chosen in range [-200,200]

// load the matrix into the texture
LoadFloatTexture(textureName, textureWidth, textureHeight, numberOfChannels, randomMat);

Finally we read the contents of the texture into another OpenCV Mat and compare it with the original Mat: 

cv::Mat outputMat = ReadFloatTexture(textureName, textureWidth, textureHeight,
                                     numberOfChannels);

cv::Scalar sumOfDiff = cv::sum(randomMat - outputMat);
MyLOGD("Sum of difference of entries in 0-th channel = %f", sumOfDiff.col(0));

As you can see, it is fairly easy to treat a texture in GLES as a 2D array of floating point values.

Set texture as output of rendering call

Finally let us see how to render to a texture. We will carry out the initial setup in this tutorial since it will help to test if your device can support rendering to a texture.

At the end of the function InitLoadReadTexture in floatTextureClass.cpp, we have the following code: 

glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
                       GL_TEXTURE_2D, textureName, 0);
CheckFramebufferStatus();
// set the texture as render target
GLenum attach_mode = GL_COLOR_ATTACHMENT0;
glDrawBuffers(1, &attach_mode);
CheckGLError("InitLoadReadTexture");

This code is fairly similar to ReadFloatTexture in texture.cpp. We begin by attaching the texture to the FBO at GL_COLOR_ATTACHMENT0. Then we check for framebuffer completeness. Finally we indicate to GLES in glDrawBuffers that the output of fragment shader should be written to the 0-th attachment point of the framebuffer.

If this code executes without any errors, i.e., both CheckFramebufferStatus and CheckGLError return without error, then we should be able to render to a texture on the device.

Summary of the code

This project is fairly simple and has many similarities to previous tutorials. FloatTextureActivity is the only activity in the app and it has a AsyncTask that checks if the device supports OpenGL ES 3+. Once the GLES surface is created in Java, we call a native function to generate and setup a framebuffer object for offscreen rendering. Then we initialize a floating point texture, load it with values from a randomly generated OpenCV Mat, and read back the contents of the texture into another OpenCV Mat. In order to test the capability of the device, we also setup a texture as the render target.