What makes the biggest difference in sound quality?

there could be an open forum for say, car guys. You could have a group of these car guys talking about how they reduced their best lap times by seconds while outlining exactly what they did to their vehicles to lower their best lap times. Another group of car guys come along to tell the first group they are all nuts and that none of their tuning tips work, some even claim to have tried their tips and that none of them lowered their lap times. The reality is, both of these camps could be correct. The problem is perspective and details. The drivers reducing their lap times could be Porsche and Ferrari pilots while the second group could be Miata drivers. Knowing the details of an audio system is extremely important to put anyone’s perspective in perspective.

As to cable length, shorter is almost always better to a point, but if you must have long cables, make them your interconnects and not your speaker cables.

If we can start with consistent times, then that is usually better than occasional low times.

At least we can measure the track times.
Here it is like we describe how the car feels.

In my experimentation and experiences over several decades, those equipment pieces that do the transduction are the most critical for sonic quality.  Turntable/arm/cartridge, CD/Player/transports and speakers come to mind. for me as having the most-significant effect on sonic quality.  Of course, EVERY piece that touches the digital or analog signal will affect the sound.  It's just degrees of it in my opinion.  One other important aspect of listening is the listening-environment !  Much over-looked, it can make the difference between happy listening and frustration.  Make sure you take care of your acoustics !  Most likely the least-expensive "component" change you can make for significant sonic accuracy...  

I didn't take the time to read all of the many responses in this thread, but I did review a few.

I'll throw in my recent experiences with improvements to my system which have all been derivatives of trouble shooting nagging and continuing problems.

The first problem I was having was with overheating at the power amp/amps.   I could not get a handle on what was causing this problem.   After a lengthy discussion with a buddy of mine who was a Bell Labs electrical engineer, we concluded that I was probably having a voltage sag under heavy load at the outlets for the amp.   He asked what material my main service entrance was made of and I told him the rails were aluminum.  The first thing he said was that the electrician who installed the service entrance should have known not to use an aluminum panel when I am 1000 feet from the ocean.  The oxidation resistance on the rails is most likely causing the issue.   I had the service entrance changed out for a copper rail panel.   Problems solved and what a nice improvement in overall performance and I have not had a heat problem since.

The next change which was jaw dropping was after I purchased a pair of Anthem M1 mono block Class D amps which initially were running on dedicated 120V mains and were changed over to dedicated direct 10 ga. feeds from the service entrance and 240v each.   That was simply amazing in the ranges the amps would play, perhaps it was due to the amps now being able to play at 2300 watts each at 240V.

Next up:  for streaming music (I am usually a two channel vinyl guy), I use my Sony XBR TV with the YouTube music service feeding an Anthem AVM60 preamp through the Audio return line and had been using a decent Belden LE series HDMI which had worked just fine for a long time.   I have no idea what happened but last summer I started to have a lot of handshake problems between the AVM60 and the Sony (I suspect there was an automatic update done one night)... it was becoming very hit or miss if the two would even connect.  When they did and usually after doing all kinds of resets, things were fine.  One night that was it, no connection and it never came back.   Tech support at Anthem and Sony worked with me on everything, nothing worked.   One person said I should maybe try another HDMI cable, I had a few on hand, all similar to the one I was using and shorter lengths, still no go.

I decided to try a high end cable so I shot over to Best Buy and picked up an AudioQuest Carbon 48K silver plated cable.   I swapped it in and voila!  Instant handshake, snappy functions on the TV and the frequency range on the upper range really opened up, it was so evident that I thought I probably had turned up the treble setting by 3 db by accident, but no it was still set on flat.   It was so impressive that I went back over and picked up the Vodka 48K cable to a/b it for fun with twice the silver content and the upper end picked up a huge increase in openness.   The Carbon went back to the store, the Vodka remained in place.

Last item:   I just received my 4th new subwoofer and while doing the set up and phasing I decided to play with the crossover settings on the system.   I put all of the subs at 150hz crossover so that they will play the entire bass range with no upper end roll off.   I went into the bass management of the Anthem and put the main speakers at 80 hz. crossover to roll off the bass under 80 hz so that they would play the mid/upper bass ranges and defeated the factory default settings on the LFE and set it at bypass to put the full unaltered (I don't use any ARC room correction at all) bass range through to the subs.   WOW, I have never heard such musical, articulate bass.  It's as if the bass guitarist is in the room.

Hopefully my learning experiences above can help out for anyone who has not considered any of the above.  

Room.

Interestingly, the source seems to be be most important part.  Folks focus on the tweaks, but the turntable, CD player, etc seem to make the most difference.  There wasn't much talk about the source in the comments.

The room makes a huge difference with a lot of high end equipment, so I am not surprised that that came up a lot.  Some equipment, such as McIntosh, is designed to be more flexible with unideal room setups.  If you have an unideal setup, make sure you audition your equipment in your room before you decide.  That was probably the most expensive mistake I've ever made.

Next would be the speakers and how they match up with your amp.

Then the preamp.

Then the interconnects and speaker wires.  One thing I found was cost is not equal to sound quality.  For example, I use a very expensive interconnect between my CD and preamp, but the same cables sounded horrible between my amps and preamps.  A much cheaper set of interconnects worked much better there.  I still really don't understand what works best, but I've found through trial and error there can be large improvements between different interconnects.  Anticables have worked well in most situations, esp speaker wires.

Other tweaks can make a difference once you figured out the above.

A power regenerator is great with a turntable.

If you do vinyl, a good record cleaner is worth its weight in gold.