Well, ok, that's just about exactly what it is. You've identified what "it is", as well as your current level of attachment/attraction to it. Nothing really "difficult" about that, eh? More like the act of a mature "listening artist" (apologies to Elder Dahn). What can, and often does, change is how far into that rarefied bubble you want to go/feel comfortable going at any given juncture of your life. And that is a function of your life, which is, as they say, subject to change without warning (and boy howdy is it ever...), not of the music, which by nature of it being a fixed quantity (i.e. - a recorded document/performance of certain specific people doing a certain specific thing at/in a specific time/place) is what it is.
So really, the question you're asking is a fair one, but any insight gleamed from the answers is inevitably going to be about the individual who's responding, not about the music itself. A corollary set of questions that might provide a more provocative range of responses would be to ask to purveyors of generally-perceived "difficult" musics why they do what they do, does it bother them that their music is peceived as difficult, do they feel that they "should" have a larger audience than they do, is their "message" one that can only be delivered in one specific manner, and who, if anybody, do they blame when that message is not successfully conveyed?
I think, based on both perception and experience, that a lot of people who provide us with this "difficult" music don't really, as a matter of principal, give too much of a shit about its "difficulty", and that is how it should be. Why they don't give too much of a shit is where things open up into some pretty interesting and widely diversified territory, not all of it necessarily "healthy", yet some of it being gloriously so.
For myself, if I'm not interested in the what - the music - I'm probably not going to be intested in the "why". If I were interested in the "why", I'd study philosophy, and that's probably the last thing I'd do in the lifetime I have.
That said, my reactions to the fixed quantities of various recorded musics have changed many times over my lifetime, as my life has changed (with or without warning).