The question is always where on the slope to you remove freedom of speech.
If I started going on about 'Kill all Jews' or 'Kill all Blacks' - would it be pointless to bother with me? I have no power, no followers, nothing. But if say this comedian starts taking his hatred to the next level, do you stop him from talking? Who knows ...
As far as I know, the FA is the only establishment on earth that takes into account numbers of followers when deciding to hand down an increased punishment. If you repeatedly abuse your free speech, the punishments should be escalated with each offence and there will come a point where the judge has to say I gotta stop you there, as the next time it's the chair for you. In that circumstance someone's right to free speech should be removed for their own good by any means necessary.