Friday, February 26, 2016

Mask Layers for Blender

When you are rendering out your render layers for compositing, you may run into the issue of layers that are improperly overlapping (particularly if an object on one layer moves in front of objects on the other). This is where the "Mask layers" come in handy.

I have a scene with 2 cubes and I want the cube that is in front to move behind the cube that is in in the back. I want my image sequence to look like this when I begin to work in the "Compositor":
Before
After

In the RenderLayer section of the properties panel, the layer that is originally in the back (ex. Cube 1) needs to add a mask for the layer that is in front but moves behind (Cube 2).

If you look at the setup below, for Cube 1, I've added a "Mask Layer" with the Cube 2 layer selected.
Renderlayer 1
Renderlayer 2

The result looks like this:

Before
After
Without the "Mask layer" enabled, I would have ended up with some unfortunate overlapping at the end of the image sequence:
Before
After
It is best to use as few mask layers as are needed. Whenever possible avoid having 2 layers mask each other out.  If two layers mask out each other, a thin transparent "halo" or border results around the edge of each object. In a simple scene, adding a "Translate node" between the object(s) you want to move and the "Alpha Over node" can be enough to hide that halo. In my case I needed to set the x-axis to -1.0 and the y-axis to 1.0. In some circumstances, a "Scale node" or a combination of a "Scale node" and "Translate node" may be needed. Keep in mind these fixes only work in very simple scenes.

Before
After