Question about length of SPDIF COAX interface cable length

grm,  I'm not sure if there is any formula, and case of your 8" cable shows it.  What counts into calculation of transmission line effects is transition time that can vary between 5ns for dedicated transports to over 50ns for average gear.  In addition the highest slew rate is usually at the beginning of transition (first knee).  When calculating trace length on PCB it is a little easier, because logic family is known, PCB material is known, etc.
At the end, result is what counts and your ears are the best judge.

I would choose 8" for sure.  Cable becomes "transmission line" when propagation time (one way) is more than 1/8 of transition time (rule of thumb).  Transitions for typical transport are about 25ns.  Allowed travel time is about 25/8=3.125ns, equivalent to about 0.6m (assuming 5ns/m speed of electricity in the cable).  In reality it should be much less, because it includes length of connections inside of gear (including PCB traces) on both ends (source and DAC).  I would assume 12" to be maksimum, so 8" should be safe.  Designers use the same method to decide if traces on PCB need termination (impedance matching).  

When you need/have longer cable reflections inside (on characteristic impedance boundaries) can modify (disfigure) transition edge affecting timing (threshold point), causing timing jitter.  Threshold is roughly in the middle of transition at 12.5ns from the beginning of transition (knee).  It is equivalent to 2.5m travel both ways (to the end of the cable and back) equal to 1.25m cable.  1.5m cable should be long enough for reflections from the end of the cable to miss threshold point.   

grm, I'm glad 8" works for you.  It is safer not to get reflections to start with then trying to manage them, IMHO.

I tried it with .5M, 1.0M, 1.5M cables on my own DAC and prefer the 1.5M as well..

I would also pick 1.5m from these choices.  8" (0.2m) shouldn't have reflections at all (not a transmission line).

That's surprising.  Brightness would suggest a presence of the jitter (that translates to added noise).  What about clarity/transparency and imaging?  It is remotely possible that some electrical noise is injected from the source, possibly thru the shield, while longer cable, having more shield inductance, impedes electrical noise injection more, but I'm grasping at straws.  Did you keep both gear pieces in the same physical location for both tests?  Did you notice it at the same time of the day?  Often people experiment after work and dinner around 6-7PM, when radio stations power is suddenly reduced by FCC regulations (better signal propagation at night).

grm,  I assumed typical 25ns transition time, but perhaps it is different.  Maybe your source is swinging faster, especially at smaller capacitive load (shorter cable).   Whatever it is - ears are the best judge.  Same goes for different 1.5m cables - cheap cable can have by chance better impedance matching with your gear, than very expensive one, resulting in better sound.  

@jjss49   I tend to overanalyze things.   I'm sure nobody decided upon 1.5m based on electrical calculations, but rather on empirical experience and calculations came later to find explanation for that. 

Uberwaltz, In case of regular cable coiling doesn't matter, since cable is not a wire but the pair of them with opposite direction currents, that produce opposing magnetic flux.  That would be a common mode choke, and that's how non inductive bifilar resistors are wound.  I think, that magnetic field on the outside of the coax cable is zero, otherwise coiling coax would change its inductance, resulting in change of characteristic impedance (being roughly SQRT(L/C) ) and we would have huge, noticeable problem.
You can find similar conclusion here:

https://pages.uncc.edu/phys2102/online-lectures/chapter-7-magnetism/7-3-amperes-law/example-magnetic...

I don't think that coiling coax creates any antenna.  The fact the people usually coil coax to >12"dia. is likely only to avoid mechanical damage (often multiple layers of shield).