Seems like a false choice. There is more than enough music that I really like that is well engineered and sounds great in my system to keep me satisfied. I've never found a need to listen to music I don't like. Or that is a terrible recording.
The music I like is more enjoyable if it is well engineered and reproduced on a quality system as opposed to poor system. A lot of folks are perfectly happy listening to music on a $100 bluetooth speaker and that's great if it works for them. But the idea that if you like high quality equipment you must not be an audiophile doesn't make much sense.