What is Inversion of Control?

Inversion of Control (IoC) can be quite confusing when it is first encountered.

1. What is it?
2. Which problem does it solve?
3. When is it appropriate to use and when not?