It's said that when Nat Pierce was Woody Herman's pianist and band manger, he gambled away the band's two week payroll one night in Vegas. Had to work with Herman a year longer than he planned to pay it back.
But it a non-musician named Abe Turchen who really did Herman in. When Turchen was the band manger he gambled away the band's withholding taxes over a number of years. Herman didn't know what was happening and was later held liable for all of it, since the band was unincorporated. Almost put Herman on the street. He could never retire because he had to keep the band working just to put a dent in the debt (which had been inflated by the IRS beyond all reasonableness).