How to detect mouse over some object in OpenGL?

Question

I'm making simple 3D game. What is the best way to detect mouse over object in 3D scene?

Solution

Actually, I would avoid using these picking methods and just do it mathematically. Create a straight line from the mouse cursor position straight into your scene and intersect it with the bounding spheres of every object in the scene. For each bounding sphere it intersects, simply check which has the vertex nearest to the eye position.

The straight line can be created using this algorithm with z set to 0 respectively 1.

math::Vec3f windowToObjectf(const math::Vec3f& windowCoord) {
  math::Matrix4f modelViewMatrix;
  math::Matrix4f projectionMatrix;
  std::array <GLint, 4> viewport;
  glGetFloatv( GL_MODELVIEW_MATRIX, modelViewMatrix.data() );
  glGetFloatv( GL_PROJECTION_MATRIX, projectionMatrix.data() );
  glGetIntegerv( GL_VIEWPORT, &viewport.front() );
  math::Vec3f ret(0, 0, 0);
  auto succes = gluUnProject( windowCoord.x , windowCoord.y, windowCoord.z, modelViewMatrix.data(), projectionMatrix.data(), &viewport.front(), &ret.x, &ret.y, &ret.z );
  RASSERT(succes == GL_TRUE);
  GL_RASSERT();
  return ret;
}

Comments

[Bartek Banachewicz]

possible duplicate of OpenGL GL_SELECT or manual collision detection?

[Bartek Banachewicz]

What benefit does it have over something like encoding entity IDs in color?

[Nicol Bolas]

@BartekBanachewicz: You mean, besides the obvious like not having to re-render the entire scene? And not inducing a GPU->CPU read-back?

[Bartek Banachewicz]

We're reading 4 bytes tops, does it really matter? I'd have to measure though.

[Bartek Banachewicz]

That's subjective, IMHO. I am not saying this code/approach is wrong! I wrote similar over this weekend for my project. :) I just think that color encoding is simpler to comprehend, and with some optimizations (like rendering only one pixel) gives amazing results with great ease.

[JasonD]

The code presented here is only "easier" because it omits the important part of actually intersecting the generated ray with the scene. The code for rendering a single pixel and reading it back is hardly complex or "hard to handle". There are pros and cons to both techniques, but "code complexity" is hardly a concern.

[Viktor Sehr]

@JasonD: I'd say it's almost always more complex and error-prone to use GL for things you can do off-GL, because of all states in GL you have to handle.