How to Avoid Duplicate Content Penalties

We've all heard of duplicate content and most webmasters know that it's a negative step to use it on a web site.
But what exactly is it and why is it bad?