I apologize for ruffling any feathers, but IMHO he often comes off as an "elitist" type, trying too hard to be hip (I feel the same way about Dick Cavett). I read his (Allen's) book Dumbth, which was his commentary on the "dumbing down" of America, and to me he kind of came off like a jerk.
I do recognize his love of jazz and appreciate his efforts to further jazz on his shows, though!
Just one person's opinion, that's all.