OpenGL C++ mouse ray picking glm:unproject

Question

I'm currently working on a C++ game engine and I want to build mouse interaction in the application. I've done this previously by ray picking, but back then I used a fixed mouse position and now I want to do it without. I read that you can use the glm::unProject function to do this, but mine just doesn't work. The coördinates this function gives aren't right. What am I doing wrong?

rscore_projection_matrix = glm::perspective(45.0f, (float)(windowWidth)/(float)(windowHeight), 0.1f, 1000.0f);
rscore_view_matrix = glm::lookAt(glm::vec3(lengthdir_x(16, rscam_direction)+rscam_x, rscam_z, lengthdir_y(16, rscam_direction)+rscam_y), glm::vec3(rscam_x, 0, rscam_y), glm::vec3(0,1,0));
rscore_model_matrix = glm::mat4(1.0f);
glm::vec3 screenPos = glm::vec3(rscore_mouse_x, rscore_mouse_y, 0.1f);
glm::vec4 viewport = glm::vec4(0.0f, 0.0f, windowWidth, windowHeight);
glm::vec3 worldPos = glm::unProject(screenPos, rscore_model_matrix, rscore_projection_matrix, viewport);

I use the vec3 worldPos positions to draw the object.

Solution

Not sure if this would help you, but i implmented ray-picking (calculating ray's direction) this way:

glm::vec3 CFreeCamera::CreateRay() {
    // these positions must be in range [-1, 1] (!!!), not [0, width] and [0, height]
    float mouseX = getMousePositionX() / (getWindowWidth()  * 0.5f) - 1.0f;
    float mouseY = getMousePositionY() / (getWindowHeight() * 0.5f) - 1.0f;

    glm::mat4 proj = glm::perspective(FoV, AspectRatio, Near, Far);
    glm::mat4 view = glm::lookAt(glm::vec3(0.0f), CameraDirection, CameraUpVector);

    glm::mat4 invVP = glm::inverse(proj * view);
    glm::vec4 screenPos = glm::vec4(mouseX, -mouseY, 1.0f, 1.0f);
    glm::vec4 worldPos = invVP * screenPos;

    glm::vec3 dir = glm::normalize(glm::vec3(worldPos));

    return dir;
}

// Values you might be interested:
glm::vec3 cameraPosition; // some camera position, this is supplied by you
glm::vec3 rayDirection = CFreeCamera::CreateRay();
glm::vec3 rayStartPositon = cameraPosition;
glm::vec3 rayEndPosition = rayStartPosition + rayDirection * someDistance;

Explanation:

When you multiply vertice's position with View and Projection matrix, then you get pixel position. If you multiply pixel position with inversion of View and Projection matrix, then you get world's position.

Though calculating inverse matrix is expensive, i'm not sure how glm::unProject works, it may do the same thing.

This only gives you world-oriented direction of ray (and you should already have camera's position). This code doesn't do 'collision' with objects.

Rest of te code of camera's class is here.

More info can be found - in example - here.