As for the whole "rules" thing...
As somebody who, like Allen, plays "both types" of music and enjoys doing so (and who has also known a fair share of avant-garde "posers"), I think the issue is not so much intentionally breaking the rules for the sake of rebellion as it is enjoying the freedom to use what is needed to tell a particular story, to disregard what is not (nothing creates a bog faster than doing something you don't need to do jsut because indoctrination insists you must), and to do this in any combination/ratio to meet the needs of any and all situations/moments.
When I was a student, the mantra was "You have to learn the rules before you can break them", which seemed at once a mixed message, as if the need to break the rules was already acknowledged, but "they" wanted you to learn them from "them" so they could have a gig, and because "they" still believed in their heart of hearts that the rules didn't need to be broken after all. Too much math for R&B...
My mantra is this - "You should learn all the rules so you can decide when to use them." Because sometimes you do want to use them. But sometimes you don't. It all depends.
This isn't an act of willful rebellion or sociopathy or anything like that. It's just an embracement of the full freedom that I believe that we as expressive creatures should have, DO have, at our disposal. But it is a freedom that I believe comes with responsibility, a responsibility that mandates that we be honest with our expression, and that we use every tool at our disposal as we create these expressions. This doesn't mean that we have to use every too (i.e. "rule") at every juncture, but it doesn't mean that we willfully not use something that would work for us either.
If seeking a fuller realization of one's expressive capacity is "rebellion", it's a rebellion only against structures/strictures that have been imposed, intentionally or unintentionally, benevolenty, malevolently, or otherwise, on humans by other humans. I don't consider that a rebellion, I consider it working out terms of a relationship.