A penalty applied directly to the ranking of certain pages with certain keywords, typically after a breach of Google's guidelines.
It can be difficult to detect when you've been penalised, as most actions are taken quietly. Best way to notice is to keep an eye on your traffic, and look out for any unexpected, drastic drops.
Penalties typically only occur when clear breaches take place, such as duplicate content and plagiarism.