I don't know, maybe i'm a misogynist, but I really don't see harm in that joke either (thought it was more flirtacious than anything). His comments the other day were ill advised and silly (though I found Key's tone when he said "do me a favour love" a lot more offensive) but I really don't think Andy Gray is a sexist on the basis of 2 clips.
Maybe he really is, as again, I don't know how he conducts himself behind the scenes.