One persons rights cannot overrule another's. You chose to smoke, we didn't. I do agree it shouldn't be banned because prohibition of anything is silly unless it's something extremely dangerous.
I would have to question whether or not second hand smoke is or isn't extremely dangerous. Using your logic smoking would have to be banned if found to be extremely dangerous. So i guess thats where I think the arguement lies: a person's definiton of "dangerous".