Can I try both and just use the best sounding one?
That's exactly what I would recommend that you do.
There may be no technical reasons not to try both, but there are technical reasons that will affect the sound. First, power delivery will be better with one of the taps, ie more power into the load, than with the other. Second, the 4 ohm tap will have half the output impedance, which means twice the damping factor, and so will interact with the load differently, affecting both frequency response, due to the voltage divider interaction with varying impedance of the load with frequency, and bass quality due to more damping.