What does listening to a speaker really tell us?

Ok. I got lots of advice here from people telling me the only way to know if a speaker is right for me is to listen to it. I want a speaker that represents true fidelity. Now, I read lots of people talking about a speakers transparency. I'm assuming that they mean that the speaker does not "interpret" the original source signal in any way. But, how do they know? How does anyone know unless they were actually in the recording studio or performance hall? Isn't true that we can only comment on the RELATIVE color a speaker adds in reference to another speaker? This assumes of course that the upstream components are "perfect."