That could be due to the surface you've got it on.

If it's a shiny surafce, or a laminated mousemat or whatever, then it does affect it and, as you say, make it spin in random directions.

I use a plain wooden surface, that's smooth as a you'll ever get it, and it works fine. It anythin does start happening then try it on your leg or an excercise book type thing. Works a treat.