It is most likely a combination of overloading the room and your ears (and possibly the equipment as distortion generally increases especially with speaker designs)
Ears will also "compress" as they compensate for the loud volume. This can be called "ear fatigue" where our ears are no longer sensitive to subtle dynamics.
And probably the biggest one is the room acoustics being overloaded. If it takes X milliseconds for a certain frequency to decay to Y decibels, if it's super loud, Y decibels will take longer to reach because of the sheer volume which will cause distortion and phasing as the frequencies at play are interacting with each other for a longer period and at a higher volume.
I've found for low volume listening tube amps are great for this reason - more full bodied at lower volumes. Start to crank it though and the solid state designs around the same price seem to outshine them. Just an added two cents. I'm sure this isn't always the case (I've never tried a Decware and I hear they are better with this) but just in my experience YMMV