I agree with you up to a point, but languages are always changing, which can lead to multiple uses for words. I know plenty of people who use gay in that context, but are not homophobic.
You could also say calling people 'retards' is insulting to mentally retarded people too, but people don't...