You know, you are 100% correct.
I don't think I appreciated him enough throughout most of his career, because FM radio shoved his music down our throats. But stepping back and looking at the bigger picture, I can't help but agree with you.
It's odd. When Bowie died I suddenly had an urge to go back and revisit his career/catalog because I thought I'd truly missed out on something special. And while he DID put out some great tunes, I found overall that my opinion of his career really didn't change. Not so with Petty. Been hearing some of his older stuff on the radio today, and realizing that, yeah, I should have been paying closer attention.