There's a few places that I really enjoy shopping at. On top of the list is:
Paul's CD's in Pittsburgh (hands down, the best jazz stock around, and a great used section)
Exchanges throughout the area of Pittsburgh and Cleveland (found plenty of gems at decent prices)
Half Price Books (another great place to frequent when the occasional gems come rolling through, like when someone brought in 3 boxes of jazz cds)
And that's about it. All the Borders in the area are terrible and B&N are overpriced. Its really sad if you like the old brick and mortor stores, and really don't care to shop on line. Its becoming a thing of the past.