Rotate Normals in Shader

Question

I have a scene with several models with individual positions and rotations. Given normals, the shaders apply simple bidirectional lighting to each pixel.

That is my vertex shader.

#version 150

in vec3 position;
in vec3 normal;
in vec2 texcoord;

out vec3 f_normal;
out vec2 f_texcoord;

uniform mat4 model;
uniform mat4 view;
uniform mat4 proj;

void main()
{
    mat4 mvp = proj * view * model;

    f_normal   = normal;
    f_texcoord = texcoord;

    gl_Position = mvp * vec4(position, 1.0);
}

And here is the fragment shader.

#version 150

in vec3 f_normal;
in vec2 f_texcoord;

uniform sampler2D tex;

vec3 sun = vec3(0.5, 1.0, 1.5);

void main()
{
    vec3 light = max(0.0, dot(normalize(f_normal), normalize(sun)));

    gl_FragColor = texture(tex, f_texcoord) * vec4(light, 1.0);
}

For objects without rotation this works fine. But for rotated models the lighting is rotated as well which, of course, shouldn't be the case.

The reason for that is that the normals are not rotated. I already tried f_normal = model * normal; but this applies both rotation and transformation to the normals.

So how can I rotate the normals in the vertex shader before I send them to the fragment shader for lighting? What is a common approach?

Solution

You do need to transform the normals, by the upper 3 rows/cols of the model-view-projection matrix. (If you are performing any scaling though, you need to use the inverse transpose of this matrix. See this article).

mat3 normalMatrix = mat3(mvp);
normalMatrix = inverse(normalMatrix);
normalMatrix = transpose(normalMatrix);
f_normal = normalize(normal * normalMatrix);
// You should also send your tranformed position to the fragment shader
f_position = vec3(mvp * vec4(position, 1.0));

In your fragment shader, you need to calculate the distance from your light source to your fragment and normalize. Find the dot product of the normal and the light vector and multiply this by the light color.

vec3 light = normalize(sun - f_position);
light = max(dot(f_normal, light), 0.0) * vec3(1.0, 1.0, 1.0);
gl_FragColor = texture(tex, f_texcoord) * vec4(light, 1.0);

There is definitely room for optimization in my code.

I recommend this book OpenGL 4.0 Shading Language Cookbook.

Comments

[danijar]

Thanks. Would you advice me to send a separate normal matrix (including only the model rotation but neither scaling nor transformation) to the shader? I saw that practice somewhere and wonder if that is better than re-extracting the rotation from the model matrix in the shader.

[bwroga]

Sorry about that, my time ran out while editing the previous comment... I think the scaling is necessary, because it can change a surface's distance to the light source. And since you need the scale, you need to correct for scale by using the inverse transpose of the upper 3 x 3 matrix. I do send my normal matrix to the shader as a uniform though, so that it is only calculated once instead of once per vertex. You might need a math library to do this though, since your platform might not provide functions to perform these matrix calculations.

[danijar]

That makes sense. By the way I use GLM.

[bwroga]

More importantly, the scaling is necessary because it can change the angle of the surface to the light.

[bwroga]

Edit: Regarding my 2nd comment, scaling the normal actually does not have anything to do with distance. Distance only applies to the fragment position, not the normal. The normal is only scaled for the reason mentioned in my 3rd comment.