I understand what you are saying.
I typically listen at fairly low volumes to most jazz and world music. However, when I listen to orchestral pieces I typically carefully dial in the volume. I started this when I was attending symphonies every week and I would hear a symphony that started with some tiny sound coming out to of the background ambience and with crescendos at... well in the symphony hall, a bit too loud (causing ear overload). I'd carefully calibrate so at home I could hear the first instrument just after its started and the creshendows would not overload my ears... Maybe what you are calling flattening... your ear is like overloaded with stuff so detail gets scrambled.
I made note of this volume (yes, I know it varies across recordings) but for most it puts me right there. The is my "live" setting.
My system doesn't loose the dynamic range until way beyond what I find comfortable. That level is very system dependent. Or maybe that if the flattening you are talking about.