A philosophical question.

I want to pose a sort of philosophical question about our listening to music.
The obvious answer to the question is that we should listen to whatever we damn please. But the query is: should we be happy listening to our favorite composers and compositions, or should we feel guilty about not exploring new horizons and music we’re prone to hate?  For me, the obvious bitter pills are such as Liszt, Neilson and Bruckner, not to mention the Second Viennese school.  We run the risk of close-mindedness by ignoring that which we don’t know and missing out on what what glories might be out there.  On the other hand, we only have so much time, and there is a universe of more accessible music available.
I just wonder if this dilemma has crossed anyone else's mind.