Many recordings are very bright. A system flat to 20 kHz will (IME) result in fatigue. That will be accentuated by bad room acoustics (slap echo, early reflections).
Depending on one’s hearing, one might need to roll off the response to make many (but not all) recordings enjoyable. That usually means adjusting toe-in or using an equalizer. The advantage of the latter is that it can be adjusted to suit the recording.
Other sources of fatigue have been mentioned: ill health, impending hearing loss, HF distortion, or speakers with peaky or rising HF response.