Actually, you both have valid points.

The fact is, Final Fantasy Tactics and X-2 both contain and revolve around core Final Fantasy elements. Chocobos, Moogles, series-specific art signatures (like the standard Black Mage design), character classes, spells, weapons, you name it. Both games use these things frequently, and as such, share a very strong tie to the Final Fantasy universe beyond the title.

On the other hand, what is there to make of a game like Vagrant Story? That particular title is set in Ivalice, the exact same world as every FF Tactics game and both FFXII titles. However, Vagrant Story shares nothing in common with Final Fantasy beyond the world in question. Just because its set in a pre-determined Final Fantasy world, does that make it a Final Fantasy game?