I'm pretty sure this particular territory has been visited and re-visited many times and this thread will not be the last.
For me, music is melody, harmony, rhythm, emotion, timbre, tonal color, density, dynamics, etc. Its sonic presentation is secondary to these factors as well as being secondary to my enjoyment. There are many musicians who don't particularly care about sonics-- their focus is the music, not the sound.
Music is the cake and sonics are the frosting. Cake without frosting is not as appealing but whereas I can live just fine without frosting, life without cake is unimaginable. Still, it's not an either/or proposition-- it's a spectrum.
I consider myself a music-lover who enjoys good sound, rather than an audiophile.
This means that while I cannot relate to the OP's circumstances, I also choose not to listen to painfully poorly recorded/mastered CDs.
Each person eventually discovers their own natural point of balance, no?
I continue to wonder whether where one locates one's self on this spectrum is affected by whether one plays an instrument or not...