What does a 'refined sounding' speaker mean ?

It probably doesn't call attention to itself or any part of the audio spectrum, does everything effortlessly and naturally.

We can guess what it means, or we can turn to the great audio reviewer Lewis Carroll who explained it so clearly in his great audiophile tome Through the Looking Glass:

"When I use a word," Humpty Dumpty said, in rather a scornful tone, "it means just what I choose it to mean—neither more nor less." "The question is," said Alice, "whether you can make words mean so many different things." "The question is," said Humpty Dumpty, "which is to be master—that's all."

Merriam - Webster defines it as precise, exact.  So in a speaker I would say it means lower than normal distortion while remaining musical, not analytical.

 It seems that this term is used a lot in comparison between cheaper and
more expensive gear to justify the price different.

Context is everything. Within the context of comparing cheaper and more expensive gear refined definitely does have meaning.

Although first I'm going to assume we agree its really sound quality we're talking about, and that "cheaper and more expensive" are stand-ins for worse and better sound quality.

Take any one seemingly simple thing like a drummer hitting a cymbal. It goes "ting" right? Simple enough. Which is how it sounds from most budget gear. I mean decent budget gear. Crap budget gear gets it more like, "tss."

In reality the "ting" is not so simple. Depending on the brand of cymbal, where it was hit, and how hard, what stand its mounted in, and how well it was recorded, that cymbal is gonna put out all kinds of harmonics across a whole range of frequencies- and with not only a range of frequencies but a whole range of dynamics. Not "ting" but more like TiiinnNnNnnnnNggggggg taking like forever to trail off into nothingness. 

Got it? Cheap gear, ting. Coarse. Expensive gear, Tiinnnnggggg. Refined.

Totally worth it.

"Cheap gear, ting. Coarse. Expensive gear, Tiinnnnggggg. Refined."

Best explanation yet :-)