Simple, if he does not have custody, file a kidnapping charge. What he's technically done is kidnapped the boy if he doesn't have custody of him.
Beat me to it.
He's obviously avoiding you and attempting to keep the child. It doesn't sound like he's going to be interested in compromise, so have your mother file a kidnapping charge and watch how open he becomes to talking when facing a standard 10-year prison sentence for kidnapping.