"Costume" would be defined as clothes not typically worn by said musician while at work or in everyday life. Thus, top hat and tails would not be a costume for Fred Astaire, but probably would be for most everybody else. Overalls would not be a costume for Art Blakey since he really wore them (at least on that Keystone Korner album cover), but most certainly would be for Wynton Marsalis.
Here's a couple of examples with which to begin: