As a semi-local, I would most recommend Normal's for used vinyl shopping, Sound Garden for new releases on CD, and An Die Musik for classical (and a little jazz) CDs. Only if you have time, check out Own Guru on Lancaster just one door west from Broadway, and their "secret" location on Broadway for vinyl.
Also, the True Vine in the Hampden neighborhood off West 36th St used to be pretty cool for avant-garde CDs and used vinyl, but it moved and downsized recently and isn't as good as it used to be.