Extracts from the book Adobe After Effects 7 Studio Techniques
Every 2D compositor knows them - selections are key to our work. They basically specify what can and what can not be seen in a composition and thereby allowing to blend more layers into one image / video. There are several ways this can be done in After Effects.
a) Pull a matte: We basically create the invisible part of a layer ourselves. There are many ways to do that and you can find almost all of them in the Effects&Presets window -> Keying. From color keying, contrast keying, difference keying to all sorts of advanced methods, the result is always the same - you extract from the layer what you want (e.g. an actor) and make the rest transparent.
b) Use an Alpha Channel: Ever wondered what the Alpha channel is for? Well, it is basically defines the transparency of a layer. If you hit the Alpha channel view in a Composition window, you can see that some parts of the layer are white (visible) and some are black (transparent)...or, as it is in most cases, you can see just a pure white color. That's because the layer doesn't include the alpha channel (typical for camera videos where transparency isn't specified). Alpha Channel often comes only with CG videos, where the author already created it. You can create Alpha Channel in AAE also - just hit the Output Modules in Render dialog and set rendering option to RGB+Alpha.
c) Draw a Mask: The worst, but often a painfully necessary choice. If your actor goes off the greenscreen, if you don't have any Alpha Channel and no way to key things out, it's all there is left. Draw masks around the shapes you want to preserve thus making them visible while deleting anything around them. It is quite easy for a still shot...but it's outright hell if you need to rotoscope a mask that contains more than a few points (= anything more than 4-6 is A LOT) in a video longer than a few seconds. Believe me, I did it already and I've hated it every since.
d) Use a Blending Mode: Blending Modes ( define ways your layer is implemented in a composition. Different Blending Modes cause different effects and unless you know exactly what every single one of them does, you'll probably need to experiment a little (e.g. different types of transparency with Screen, Add, Ligten, Darken, Color Burn etc. modes).
There are of course tons of possible combinations as well as some special methods using effects (Curves, Levels, Generated or Simulated come with the Alpha channel included).