I think the reason why people dislike buying gil, is much the same they dislike gambling. Not nescessarily the act itself of buying gil, but the additional problems it creates. The most visible is gilseller/professional farmer monopolization. Anyone who got their Haidate pre-Diorite can tell you, it's not a small issue.
Also, people who buy gil often have no sense of it's value, and spend it frivolously, this may contribute to inflation, but the only solid connection between buying gil and inflation is that Professional Gillers are creating much more gil than can be destroyed by normal means. Example: Remedy NPCing.
So, first of all, I have to say that anyone who stands on the ground "Gilbuying doesn't affect you" is a complete retard. It obviously affects everyone who plays the game.