OpenCV image loading for OpenGL Texture

Question

I want to load an image (jpg and png) with OpenCV as OpenGL Texture.

Here is how I load the image to OpenGL:

glEnable(GL_TEXTURE_2D);
  textureData = loadTextureData("textures/trashbin.png");
  cv::Mat image = cv::imread("textures/trashbin.png");
  if(image.empty()){
      std::cout << "image empty" << std::endl;
  }else{
      glGenTextures( 1, &textureTrash );
      glBindTexture( GL_TEXTURE_2D, textureTrash );
      glTexParameterf(GL_TEXTURE_2D,GL_TEXTURE_MIN_FILTER,GL_LINEAR);
      glTexParameterf(GL_TEXTURE_2D,GL_TEXTURE_MAG_FILTER,GL_LINEAR);
      glTexParameterf( GL_TEXTURE_2D, GL_TEXTURE_WRAP_S , GL_REPEAT );
      glTexParameterf( GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT );
      glTexImage2D(GL_TEXTURE_2D,0,3,image.cols, image.rows,0,GL_RGB,GL_UNSIGNED_BYTE, image.data);
  }

The image is loaded, as "image.empty" always returns false

Here is how I render the scene using the created texture:

  glActiveTexture(GL_TEXTURE0);
  glBindTexture(GL_TEXTURE_2D, textureTrash);
  glm_ModelViewMatrix.top() = glm::translate(glm_ModelViewMatrix.top(),0.0f,-13.0f,-10.0f);
  glUniformMatrix4fv(uniformLocations["modelview"], 1, false, glm::value_ptr(glm_ModelViewMatrix.top()));

  std::cout << "textureShaderID: " << glGetUniformLocation(shaderProgram,"texture") << std::endl;

  glUniform1i(glGetUniformLocation(shaderProgram,"texture"), 0);
  objLoader->getMeshObj("trashbin")->render();

And finally the fragmentShader where I want to apply the texture to my geometry

#version 330
in vec2 tCoord;

// texture //
// TODO: set up a texture uniform //
uniform sampler2D texture;

// this defines the fragment output //
out vec4 color;

void main() {
  // TODO: get the texel value from your texture at the position of the passed texture coordinate //
  color = texture2D(texture, tCoord);
}

The texture coordinates are comeing from a Vertex Buffer Object and are correctly set from the .obj file. Also I can see the Object in my scene when I set the color to e.g. red in the fragment shader, or to vec4(tCoord,0,1); then the object is shaded in different color.

Unfortunately the screen stays black when I want to apply the texture... Can someone help me and tell me why is stays black?

Solution 1

From only looking at your texture loading code you are ignoring many considerations about how OpenCV lays out images in memory. I've already explained that for the opposite direction (glGetTexImage into OpenCV image) in this answer, but will recapitulate it here for the CV-GL direction:

First of all OpenCV doesn't neccessarily store image rows tightly packed but might align them to certain byte boundaries (don't know how much, at least 4, but maybe 8 or more?). If you're lucky it will use 4-byte alignment and the GL is set to the default pixel storage mode of 4-byte alignment, too. But it's best to manually fiddle with the pixel storage modes in order to be on the safe side:

//use fast 4-byte alignment (default anyway) if possible
glPixelStorei(GL_UNPACK_ALIGNMENT, (image.step & 3) ? 1 : 4);

//set length of one complete row in data (doesn't need to equal image.cols)
glPixelStorei(GL_UNPACK_ROW_LENGTH, image.step/image.elemSize());

Then you have to account for the fact that OpenCV stores images from top to bottom, while the GL uses bottom to top. This could be taken care of by just mirroring the t-texture coordinate appropriately (maybe directly in the shader), but you could also just flip the image before uploading:

cv::flip(image, flipped, 0);
image = flipped;              //maybe just cv::flip(image, image, 0)?

Last but not least OpenCV stores color images in BGR format so uploading it as RGB would distort the colors. So use GL_BGR (requires OpenGL 1.2, but who doesn't have that?) in glTexImage2D.

These might not be the complete solution to your problem (since I think those errors should rather result in a distorted rather than a black image), but they are definitely problems to take care of.

EDIT: Does your fragment shader actually compile successfully (in the complete version using the texture)? I'm asking because in the GLSL 3.30 you're using the word texture is also the name of a builtin function (which should actually be used instead of the deprecated texture2D function), so maybe the compiler has some name resolution problems (and maybe this error is ignored in your simplified shaders, given that the whole uniform will be optimized away and many GLSL compilers are known to be anything else than strictly standard compliant). So just try to give that sampler uniform a different name.

Solution 2

Okay here is my working solution - based on the ideas of "Christan Rau" - Thanks for that!

cv::Mat image = cv::imread("textures/trashbin.png");
  //cv::Mat flipped;
  //cv::flip(image, flipped, 0);
  //image = flipped;
  if(image.empty()){
      std::cout << "image empty" << std::endl;
  }else{
      cv::flip(image, image, 0);
      glGenTextures(1, &textureTrash);
      glBindTexture(GL_TEXTURE_2D, textureTrash);

      glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
      glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

        // Set texture clamping method
      glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP);
      glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP);


      glTexImage2D(GL_TEXTURE_2D,     // Type of texture
                     0,                 // Pyramid level (for mip-mapping) - 0 is the top level
                     GL_RGB,            // Internal colour format to convert to
                     image.cols,          // Image width  i.e. 640 for Kinect in standard mode
                     image.rows,          // Image height i.e. 480 for Kinect in standard mode
                     0,                 // Border width in pixels (can either be 1 or 0)
                     GL_BGR, // Input image format (i.e. GL_RGB, GL_RGBA, GL_BGR etc.)
                     GL_UNSIGNED_BYTE,  // Image data type
                     image.ptr());        // The actual image data itself

      glGenerateMipmap(GL_TEXTURE_2D);
  }

Comments

[Christian Rau]

So does it work this way? By the way, why did you change the filter and clamping modes? And why do you generate mipmaps while not using a mipmapping filter?

[glethien]

Yeah it is working this way - Surely not the best solution and has a lot of potential to improve - But it is working - And mipmaps will be implemented as the next feature, I've just forgot to remove the line

[Pete]

I am using your method to show image, but I am getting a read access violation for using image.ptr(). Please see my post here: stackoverflow.com/questions/45013214/…

[Maverick Meerkat]

I only needed glPixelStorei when using GL_UNSIGNED_BYTE, with an image width NOT divisible by 4. When I use GL_FLOAT, or, when I use a width divisible by 4 - it wasn't needed.

[Maverick Meerkat]

After digging a bit more I realised that some part of the answer is misleading. There's nothing "fast" about a 4 byte alignment. In any case you certainly have no way of speeding up anything by setting the alignment in openGL, if it's not already aligned. This parameter only tells openGL where to expect the next row of pixels. If your image is tightly packed, you should use 1. Also, row-length is only needed if you're using a subset of the image (a roi - rect of interest), or if for some reason there's extremely big padding between rows. Otherwise it's not needed.

[Christian Rau]

@DavidRefaeli Well, of course your alignment has to match your data. Just setting it to 4 when your data isn't 4-aligned makes no sense. I'd be surprised if that's what the answer suggests, though (in fact it makes it directly dependent on the data). However, using 4-aligned data (and telling GL so) is likely faster to copy, even if that's not the primary reason to set it. Your remark about the row length setting makes sense, though.

[KyranF]

thanks for your answer! It solved my image distortion after decimation filter made the row lengths non-divisible by 4, the line glPixelStorei(GL_UNPACK_ALIGNMENT, (image.step & 3) ? 1 : 4); fixed it for me. Cheers!