There used to be and may still be a junk store in North Beach that often had some good vinyl (in addition to a decent selection of reasonably priced instruments). The place is a litte chaotic but I found some good stuff there back in the day. As of about 1.5 years ago it was still there.
There also used to be a book store on Polk near either California or Pine that had a few bins that were worth flipping through. As a teen I found a ridiculously expensive Residents record in there for about five bucks, and there were usually quite a few old looking jazz records, but I was too young too know what they were at the time. I recall the owner being a jazz guy though, so it is probably worth checking out.
Recycled Records on Haight is pretty good for rock and at least used to have a decent jazz section.
Other than that, Amoeba. They killed all the others off.