I agree, but then that's the way I feel about any actor who operates this way. Heck, even Jerry Lewis was funny for a picture or two..
As a director, though, I think it's true that the 'creepy pedophile' tag did hurt him greatly. After that episode, I just didn't seem to notice people discussing his movies as much as before. People can overlook a lot for art; apparently marrying your step-daughter isn't one of them.