Using Layers and Bitmask with Raycast in Unity

Question

Unity's Raycast functions has a parameter you could use to raycast to a particular GameObject. You can also use that parameter to ignore particular GameObject.

For exmple the Raycast function:

public static bool Raycast(Vector3 origin, Vector3 direction, float maxDistance = Mathf.Infinity, int layerMask = DefaultRaycastLayers, QueryTriggerInteraction queryTriggerInteraction = QueryTriggerInteraction.UseGlobal);

The layerMask parameter is used to specified which Objects should/should not receive the raycast.

---

1.How do you raycast to a particular GameObject which is in a layer called "cube"?

2.What if you have 10 GameObjects in the scene but you only want to raycast to just 2 GameObjects and ignore the rest? How do you do that?

Let's say that those Object's layers are "cube" and "sphere".

3.What if you want to raycast to all GameObjects but ignore 1.

Let's say that the GameObject to ignore is in the "cube" layer.

4.What if you want to raycast to all GameObjects but ignore 2(multiple) GameObjects.

Again, the layers to ignore are the "cube" and "sphere" layers.

Solution 1

Most Raycast questions I see use the Layermask incorrectly. Although it works for them by luck but they usually run into issues when they actually want to exclude a GameObject from Raycast.

This answer is made to cover all those scenarios that a person would want to use the Layer to filter GameObjects when performing raycast.

1.How do you raycast to a particular GameObject which is in a layer called "cube"?

First you use LayerMask.NameToLayer("cube") to convert the layer name to layer number. The LayerMask.NameToLayer function returns -1 if the layer does not exist. You must check this before doing any layer bitwise operation.

Raycast to a particular layer ("cube" only):

//Convert Layer Name to Layer Number
int cubeLayerIndex = LayerMask.NameToLayer("cube");

//Check if layer is valid
if (cubeLayerIndex == -1)
{
    Debug.LogError("Layer Does not exist");
}
else
{
    //Calculate layermask to Raycast to. (Raycast to "cube" layer only)
    int layerMask = (1 << cubeLayerIndex);

    Vector3 fwd = transform.TransformDirection(Vector3.forward);

    //Raycast with that layer mask
    if (Physics.Raycast(transform.position, fwd, 10, layerMask))
    {

    }
}

The most important part of the example above is int layerMask = (1 << cubeLayerIndex);.

To make this answer short, I won't be checking for errors for the rest of the answer.

---

2.What if you have 10 GameObjects in the scene but you only want to raycast to just 2 GameObjects and ignore the rest? How do you do that?

Let's say that those Object's layers are "cube" and "sphere".

Raycast to the "cube" and "sphere" layers and ignore the rest:

//Convert Layer Name to Layer Number
int cubeLayerIndex = LayerMask.NameToLayer("cube");
int sphereLayerIndex = LayerMask.NameToLayer("sphere");

//Calculate layermask to Raycast to. (Raycast to "cube" && "sphere" layers only)
int layerMask = (1 << cubeLayerIndex) | (1 << sphereLayerIndex);

---

3.What if you want to raycast to all GameObjects but ignore 1.

Let's say that the GameObject to ignore is in the "cube" layer.

Raycast to all but ignore the "cube" layer:

//Convert Layer Name to Layer Number
int cubeLayerIndex = LayerMask.NameToLayer("cube");

//Calculate layermask to Raycast to. (Ignore "cube" layer)
int layerMask = (1 << cubeLayerIndex);
//Invert to ignore it
layerMask = ~layerMask;

---

4.What if you want to raycast to all GameObjects but ignore 2(multiple) GameObjects.

Again, the layers to ignore are the "cube" and "sphere" layers.

Raycast to all but ignore the "cube" and "sphere" layers:

//Convert Layer Name to Layer Number
int cubeLayerIndex = LayerMask.NameToLayer("cube");
int sphereLayerIndex = LayerMask.NameToLayer("sphere");

//Calculate layermask to Raycast to. (Ignore "cube" && "sphere" layers)
int layerMask = ~((1 << cubeLayerIndex) | (1 << sphereLayerIndex));

OR

//Convert Layer Name to Layer Number
int cubeLayerIndex = LayerMask.NameToLayer("cube");
int sphereLayerIndex = LayerMask.NameToLayer("sphere");

//Calculate layermask to Raycast to. (Ignore "cube" && "sphere" layers)
int layerMask = (1 << cubeLayerIndex);
layerMask |= (1 << sphereLayerIndex);
layerMask |= (1 << otherLayerToIgnore1);
layerMask |= (1 << otherLayerToIgnore2);
layerMask |= (1 << otherLayerToIgnore3);
//Invert to ignore it
layerMask = ~layerMask;

---

Finally, if you know the layer index/number, there is no need to use the LayerMask.NameToLayer function. Just insert that layer index there. For example, let's raycast to the "cube" layer which is in index #9. You could just do int layerMask = (1 << 9);.

See the Layers manual to read more about this subjct.

Solution 2

To add to @Programmer's comprehensive answer...

1. How do you raycast to a particular GameObject which is in a layer called "cube"?

If you want to Raycast to see if you hit a specific object only, call object.collider.Raycast(...).

object can be either the GameObject or your MonoBehaviour-derived script (this).