Lip liner can be applied both before lip colour (to keep the colour from bleeding or feathering), or after (to clean up the edges). I am a "before" person because I'm more concerned about bleeding, and if you do it right the edge will be clean anyway. "After" people are weird.
The trick to lining your lips (or eyes, for that matter), is to not try to draw it straight along all in one go. Not everyone has a perfectly steady hand, and slips happen. Doing it in short strokes is the best way to avoid accidentally drawing all over your face or ending up uneven.
Step one: Choose your liner. It doesn't have to match the lipstick exactly, usually it's good enough if it's in the same colour family. Today I'll be using Make Up For Ever "Fuschia 16C".
Step two: Sharpen. It's important that the pencil is sharp. I find that if the line is too smudgy, it doesn't create the barrier it's supposed to. I also like that sharp edge on my lips, none of that "natural blend into your face" shit. Your mileage may vary.
Step three (optional): Dot your points. To be extra sure of where your points are (especially around the Cupid's Bow and at the bottom lip), you can start with guide dots and then draw your lines between them. Nothing wrong with that. Do what you gotta do.
Step four: Short strokes.
Step five: I always trace back over the line in longer strokes to be sure it's all where it's supposed to be. In my mind, lip liner is the edges of a plastic kiddie pool and the lipstick is the water. I have no idea if this is actually true. Maybe it's not. But because my brain thinks this way, I am careful to make sure there are no leaks in that boat. Please pardon my mixed metaphors.
|Do not stop here.|
Step six: Fill. I like to overlap the edge of the liner a bit with the lip colour (especially if the liner and lipstick aren't a perfect match), but not all the way or my whole kiddie pool thing is defeated. Again, this might just be me. You can go right to the edge if that's the way you live. The colour used here is Obsessive Compulsive Cosmetics lip tar in "Strumpet".
For extra security, you can fill in the entire lip with lip liner before applying the colour over top. This helps certain lipsticks last longer, but beware: it might alter the colour slightly if your lipstick and liner don't match exactly. For example, these photos use the exact same tube of lipstick (Buxom Full Bodied Lipstick in "Menace"), but one has a plum liner that's a little too dark for it underneath, and the other has a colourless liner.
Left: Make Up For Ever "Matte Dark Plum 12C" / Right: Make Up For Ever Lip Line Perfector
Not that there's any reason not to change your colour with liner, because that's actually a pretty handy trick. Just sayin'. It's a thing. Lip liner is so much more useful and versatile than you thought, see?