upgrade what

I agree with @hilde45 get the room squared away before you start replacing gear.  Have you experimented with speaker placement and  toe-in?

What is your room look like?  System brightness is dealt with easily with a few panels.  

I would venture a guess adding a DAC will address some of your issues.  I would bypass the Denon players internal DAC's up-sampling filters and DSP.

I would check setup and room acoustics first. If you create a virtual system and put photos of your system it can help us a lot, to help you. 

If you occasionally think it is a bit bright... then it most likely is always much too bright and not fully fleshed out. 

Yes upgrade speakers. They make a huge difference.  I would recommend looking at Sonus Faber Sonnet 2, KEF Reference, Harbeth Compact... or better for these brands. 

If you want a significant improvement that is really obvious, I always budget at least 2x or more. 

Hi @bzawa – thanks for your post.  Without even seeing your room and system, I along with others think it may not be your equipment but the room’s impact on sound quality.  Here are some things to consider from personal experience:

Issue:  “a little bright and forward sounding”
Sounds like a tonal balance problem which can be a combination of decay time and frequency response mismatch to a target curve to spot when things are too long/short or too loud.  When bass decay times are shorter than those of mids/highs (i.e. due to a large open space / open concept homes), I’ve found that upper frequency reflections from untreated surfaces accentuate the brightness.  And the brightness can get worse when frequencies of about 1-3kHz are too loud relative to those below or above it as our ears are especially sensitive in this range.  A target curve can help spot “peak” offender notes above the target curve that are too loud and contributing to the brightness.  Lastly, the frequencies from a 1^st order reflection may be dissimilar in energy/loudness with too big a difference between the low midrange and high frequencies.

Issue: “lacks really good instrument separation”

This smells like an imaging issue where the singer/instruments may float laterally and aren’t locked-down so to speak. This is usually due to early reflection (0-10ms) asymmetry between Left and Right speaker reflections typically off sidewalls, floor/ceiling where sound is pulled toward the louder side.  Are your sidewall’s 1^st reflections the same distance to each speaker and constructed using the same materials?  I’m guessing ‘no.’  When early reflection issues are mixed with asymmetrical loudness between Left vs Right frequency response, the issues compound. 

Have you ever taken acoustical measurements?