Masking Brushes
Each brush has a mask that controls where and how the brush affects the landscape. These masks can be hand-painted, created procedurally or be a combination of both.
Where the mask has values equal to 1.0 (white color) the brush affects the landscape at full weight.
Where the mask has values equal to 0.0 (transparent) the brush doesn't affect the landscape.
Hand Painting a Mask
To start masking a brush, select that brush and click Initialize Paint Mask button in the brush properties. Then switch to the Mask tool.
Adjust the brush settings - radius, hardness and painted value in the tool's panel.
You can hide the preview of the mask values by clicking the Toggle Mask Preview button.
Procedural Masking
In order to create a procedural mask, you need to pass the mask through a post-process material.
That material must use Post Process mode.
Your hand-painted mask is accessible in the material as MaskRenderTarget texture parameter.