This paper engages with the recent dignity-based argument against hate speech proposed by Jeremy Waldron. It’s claimed that while Waldron makes progress by conceptualising dignity less as an inherent property and more as a civic status which hate speech undermines, his argument is nonetheless subject to the problem that there are many sources of citizens’ dignitary status besides speech. Moreover, insofar as dignity informs the grounds of individuals’ right to free speech, Waldron’s argument leaves us balancing hate speakers’ dignity against the dignity of those whom they attack. I suggest instead that a central part of the harm of hate speech is that it assaults our self-respect. The reasons to respect oneself are moral reasons which can be shared with others, and individuals have moral reasons to respect themselves for their agency, and their entitlements. Free speech is interpreted not as an individual liberty, but as a collective enterprise which serves the interests of speakers and the receivers of speech. I argue that hate speech undermines the self-respect of its targets in both the agency and entitlement dimensions, and claim, moreover, that this is a direct harm which cannot be compensated for by other sources of self-respect. I further argue that hate speakers have no basis respect themselves qua their hate speech, as self-respect is based on moral reasons. I conclude that self-respect, unlike dignity, is sufficient to explain the harm of hate speech, even though it may not be necessary to explain its wrongness.