How to make a true comparison?

In reading another post asking for observations between two turntables, a rather obvious question occurred to me - what is the best way to make a true comparison?

Solution 1 would be to use the same platform, mat, arm, cartridge, alignment, etc. on tables B and D to observe the differences, the traditional level playing field.

But what if the arm was not interchangeable and could not be fitted to both tables or, more importantly, what if a given arm was not compatible with both tables? And then the same cartridge was not compatible with the different arms?

Then Solution 2 would suggest the best comparison would be between "systems" that were optimized for each table. True, you would be hearing influences of arm, cartridge, etc. as well but you would be comparing potential VS potential.

This applies through out the audio chain. Comparing one speaker against another without driving each by an appropriate amp would not identify the potential performance and thus differences between the two speaker systems.

Perhaps we need a tilted (compensated) playing field rather than a level one to make worthwhile comparisons?
Dear Pryso, aside from some small and fairly lightweight TTs I would always suggest moving the very same tonearm/cartridge combo from one TT to the other.
Any comparism which shall give a true result or at least an idea of the differneces should be made with as few parameters changed as possible - if possible the only variable factor being the component under review.

As for loudspeakers we can only compare speakers with similar demands as for power, damping factor, sensitivity and complexity of crossover-network.

Even comparing speakers against each other driven each by a suitable amplifier will not give true results if the two amplifiers aren't identical.
We would compare speaker/amplifier-set-ups, but not speaker vs speaker.

Our dilema - its very difficult to compare any singular opart of teh audio-chain for itself.
Best chances are with TTs and preamplifiers given fairly normal impedances in teh power amplifier(s).
The problem is that what the question is usually about is "how do two components compare IN MY SYSTEM". How they would rate in a system adjusted to make the most of their virtues would be in most cases irrelevant as the person asking the question is very unlikely to have such a system already or assemble one later. What he wants to know is how components will fit into his existing system. As each part of the chain introduces different variables each system will be to a greater or lesser degree unique so the most that can be achieved is generalizations. Turntable testers use to use identical arms and cartridges, transferring the same stylus between them in MM cartridges, even this did not produce an exact identical sound. Moving the arm between tables precludes quick AB comparisons. In the end making any meaningful comparisons is a tedious job requiring considerable skill.
Well personal preferences - which are fine if its only about "fitting in MY SYSTEM" - and interaction with other parts of the audio chain simply do not allow any comparism.
Unless you imply that all other parts are "perfect" you will never be able to tell or distill the whereabouts of certain changes in sound as they might well arise from certain interactions of components linked to tiny details in their technical specifications.

From the point of logic - if there is any such thing in high-end audio at all... - we need to eliminate all other variables to conduct any test or comparism with results worth mentioning and - maybe... - transferable.

If one is going for "which of the 2(3) components is BEST in MY SYSTEM" - that is a much easier task and is always a result of "fitting into the matrix".
And even here the result is only a status quo of the moment as the results may most likely change with the next "upgrade" in the system as the synergy-effects achieved before may have gone now.

Endless story - the results depends on the goal set and on the parameters of the periphery.
Stringent or freely?
Subjective or objective?
Viewed in the context of a certain system or seen individually?
Dear Pryso: +++++ " In the end making any meaningful comparisons is a tedious job requiring considerable skill. " ++++

I agree wth Stanwall but that " tedious job " is part of a serious comparison.

Your first part of the solution 1 is the way to go: everything the same but the TT.

If you can't do it then you are not doing a straight/right/precise comparison. Even if you can do it there will be a second variable factor that is the own each TT arm board that normally is dfferent and that has influence in the final result.
The best way could be to have same stand alone ( tower ) arm board. We are talking here of what you ask: " a true comparison " where the only change is the TT it self.

Regards and enjoy the music,