I think that main problem is not what you personally said, but that they are hearing the same slur all day, every day, from different people.

Frequency they hear it is what makes it problematic.

“your personal liberty to swing your arm ends where my nose begins” but you have to be more understanding to be able and see where their nose is. Sadly, it is a bit closer than expected.