Yes, Lacy was. He was close with many postwar artists and writers.
When I was in grad school for art history, very few of my fellow grad students had any interest in or appreciation for improvised music (which I was writing about as a component of a true art history). Similarly, a number of creative musicians I've met (especially American) have had zero interest in abstract art.
The contemporary composers' world seems a bit more keyed into abstract visual art, and abstract performance art. Boulez is certainly played at massive concert halls in Europe, though he's far less well-known here. Most of the western classical/modern composition situations I attend are pretty left-field, and those people are very knowledgable. If one is attending a pops concert or whatever, I don't think one should expect much modern music to be played, just like I wouldn't expect to see the Lincoln Center Jazz Orchestra play Tri-Centric Music.
Finally, the visual art world is very much a way to move money around and it's a lot harder to funnel cash through weird music.