The answer to that is that your speaker is not 6 ohms. No speaker is 4, 5, 6, 8, or 10 ohms. That number is the AVERAGE impedance of the speaker using a non-standardized test. Listen to the speaker both ways and enjoy the best way. You cannot damage anything.
Try it both ways to see which way you like it the best, but do not make quick decissions. Let it settle in for a while. If one tap or the other has never been used it may take a while for it to break in.
I remember reading some time ago in another forum that if you had a speaker that averaged a 6 ohm load and used an 8 ohm tap on a tube amp it would run hotter than if you used the 4 ohm tap. Is this true?