Well, I guess, since "honky" is intended to be offensive, I should be offended (as a non-ethnic white guy).

But you're right; the word just doesn't seem to carry much weight.