I think every passive skill deserves at least 1 point on any kind of character unless the points are just needed somewhere else. +skills can really boost your passive skills, for instance my barbarian has 1 point into the 4 passive skills and maxed axe mastery, and the passive skills are like level 11 each or something, quite worth the 1 point.