I think that their early stuff was intelligent, unique and quite good. To me, "(Don't Go Back To) Rockville", from their "Reckoning" label, is the best song ever about being stuck living in a small, isolated town. Some of the lyrics, and the overall feel of the song, capture the experience perfectly.
Our family moved to Wausau, Wisconsin when we were all partway through school, and none of the kids liked it at all. We all counted the days until we could move away. I mentioned to my brother once "that song about Wausau" and he knew immediately, with not another word, that I meant REM's "(Don't Go Back To) Rockville." He will still refer to "Reckoning" as the album with the Wausau song on it.
At some point they became just another rock band, to me. But the early recordings can not be easily dismissed, I think.