Yes, and it's also a defiant song. The singer is saying "that's why everyone in the smart set thinks I'm a tramp", ie, because I haven't much money and prefer everyday simple activities to fashionable, sophisticated ones. "No got. So what?"
Although the versions by Sinatra, Buddy Greco and other male singers swing superbly, the song doesn't suit a man at all. It comes across as, at best, nonsense, at worst misogynous. The scene when Sinatra sings it in Paul Joey to provoke a woman is especially embarrassing. And the attempts to update the lyrics are uniformly dreadful. I'm not surprised the OP was nonplussed by the versions he's heard.
Well it makes sense for a male to sing if he's defending a woman who's been dismissed as a tramp. But IIRC in the Pal Joey film it's just confusing because Joey is trying to say that the woman in question is a tramp. (Rita Hayworth?).