Sometimes when I'm driving, my engine's rpm keeps shooting down and up whenever I take my foot off the gas.

Example:

Accelerate to 45. 1,500 rpm @ 45 mph. Take foot off gas. The rpm shoots down to 900. The car slows down dramatically, rpm goes back up to 1,100. Car doesnt slow down as fast.

Thing is this doesn't always happen, just sometimes. The car is 10 years old and has 96,150 miles on it.

What could be wrong?