How do you know what you're missing?

It's a great question!  Much of the answer depends upon what type of person are you.  If you just want to listen to music and enjoy the experience you won't really worry about how it could be even better.  The music sounds good and you're happy.  For that type of person your question is not particularly important.  There's another type of listener who is more interested in what they call "the journey".  No matter what their system sounds like, they are always thinking about the next step.  It's axiomatic that their system can always be improved.  I suspect they do like listening to music, but they really like listening to their systems.

Once you figure out what type of listener you are the answer to your question is kind of self-evident.  If you're not

Years ago ( mid nineties) when l was in the process of trying to improve my stereo system l had a chance to hear a pair of lnfinity RS lll speakers! Holy ****! they were amazing! The song that was playing was Rodger Millers King of the road. Talk about a bench mark! Another speaker that really impressed me back then was the Infinity Kappa 9's Of course I could never effort either pair back then, but by listening to those speakers it really helped me in my journey to put together a somewhat modest but very enjoyable system that l think has some of the hi fidelity traits of those great speakers

onhwy61 expresses well what one goes through 'getting there', as do the other responders.  What everyone hedges around is the old conudrum:

Age & Experience. (*groan*....'oh, No, not That again....')

What you heard when/where/how/with whom (or not)...what it sounded like that impressed, how it made the back of the neck hair respond (or whatever else occurred)....

The 'I want That daily in my life on demand' kinda thing...*s*

Then the hat trick is to Find and Keep 'That'...because once the stone is rolling it gets harder to Stop and say 'enough'. ;)

Carry a couple of cd's and/or some fav's on your phone with the appropo IC....shop around....consider what you like about the music you play and what it is that you enjoy about it....

What would seem to make it better; more discrete, dynamic, quiet in the quiet passages, nimble in the complications....

...and keep in mind the space listened within can play as big a role as the equipment...and you as well.

Only you will know the answer(s) to all of the above.

And remember:  All of us have the same equation to solve.

Good luck, and have a good time at it....which IS the point, after all...;)

very good, thought provoking question

i have a few different thoughts it spurs

1) the objective to hifi is not to make it sound like live music necessarily, it is to make it sound realistic, but 'realism' is defined in the eye of the beholder -- so you do need to hear different systems and ideally, have owners describe to you what they feel sounds good about it, and see if you gree (that's why good brick and mortar stores are so important, as well as hifi shows, and local hifi nerd friends and local audio societies etc etc)... i realize these may still be out of reach for many depending on where they live etc etc

2) it is important to try to hear live music in as many form as you can (not amplified), even a street performer singing, playing a saxophone, or a high school performance, or a church choir - remember what the sound of a live voice, a live instrument, sounds like - that purity of tone, how notes rise and decay, no harshness no grain or grit...  is what a good hifi should be reproducing

3) it is easy to fall into 2 traps -

first, that super-clarity and extreme detail is more real and good... that is a 'hifi' creation... real music is not hyperdetailed and disaggregated, it is cohesive and natural; 

second, you should listen enough to define what you like, the sound you like -- don't read reviews or watch you tube videos and let other people define that for you -- for example, some people want their hifi to sound like a rihanna or bonjovi concert... well what you hear at those concerts are huge harsh solid state amps driving huge horn loaded speakers designed to fill a stadium... i would not call that the sound of real music, but you might, and there is nothing wrong with that...

good luck and have fun

I think you are asking a very valid question. After not caring much about music for the past 40 years I’ve jumped into the whole audiophile thing with both feet. I’ve been buying and trying a variety of used equipment. So far I have 3 speakers, 4 amplifiers and 2 preamps with multiple tube options. I’m still struggling with the best way of deciding when one piece of gear is better than another piece of equipment. True A/B testing is somewhere between complicated to impossible and my audio memory only seems to last a few minutes, not long enough to switch cables around. I end up taking notes on certain passages where I think a difference between two pieces of equipment will show up as well as formulating general impressions. All the reviewers that make such definitive and detailed assertions about the speakers and amplifiers they listen to are clearly much more skilled listeners than I am or they simply make that crap  up LOL. I have been surprised at how different the amplifiers sound but I haven’t been able to choose one as being best yet. Most of the tubes in the preamp that I’ve tried over emphasize bass which I don’t like so that is easy. But in terms of what might represent the best, ultimate sound, well I don’t have a clue right now. And then there are funny things where it is hard to determine if a change makes a real difference or if you just want it to have made a difference so you hear your system as being better. I would but my Furman power conditioner into that category. Maybe it made things better or maybe I just wanted it to make things better!