Looking at all the words (I haven't heard the tune), I'm not even sure that I'd consider it "racist" - at least it doesn't appear to be racist in intent. Stereotypical, yes, but a product of its less sensitive era. As Jon correctly states, it does seem self-deprecating and tongue-in-cheek, and if an African-American singer wrote and sang it originally I don't think him using words like "nappy" and "pappy" would have given it any kind of racial connotation. That Mercer, a white dude, later sang it is a bit more problematic, but I'm not sure where to draw the line between minstrel and novelty song.
edit: Adam said it much more eloquently than me!