Definitely, Madrid is not a good place to buy jazz CDs. Over the last five years, I could count with my fingers the number of times I´ve bought in Madrid stores. And recently, Madrid Rock, where you could find a "reasonably decent" selection and a few good bargains (such as RVG for 9 EUR and the Spanish Blue Notes for 4,95 EUR) has been closed.
Like in many other European towns, you can go to FNAC, but you won´t find anything special. And prices (despite their "you can´t find it cheaper" advertisement) are quite expensive, sometimes purely crazy.
I know there a few second hand small shops, but I am not the one to recommend them (funny that I know several excellent Paris second hand stores, but no idea for Madrid). Maybe bison ravi or cayetano ???
And as for Barcelona, yes!!! go to the JazzMessengers shop. One of the best places I´ve found to buy jazz.
Cheers,
Agustín