I think whether or not he's "objectively" wrong is missing the point. You're right about language being context sensitive, it's feasible that someone can say the N word and not harbor racist feelings. For example, if you're a white person who has close friends who are black, and they have given you permission to use the word as a term of endearment, affection, what have you, then it's not really a big deal. I see that kind of thing all the time and it's fine. There's always rare exceptions to the rule that you'll find, but that's the case with pretty much everything. Given the words heavily racist implications, if you use it outside of a close circle of friends that personally know you, you're being racist because 1) People have no way of assessing your true feelings about race if they haven't personally interacted with you, so given its connotations you're pretty much going to be offending someone 99% percent of the time irrespective of whether you truly harbor racist feelings or not.