It might matter. The easiest way to find out is to simply do it. If the amp is in distress, it will protect itself. Go for it...
The other thing to consider is that the nominal impedance of your speakers is probably not the same as the nominal impedance of either the High or the Low. So one channel could be seeing a much lower impedance than 6 ohms. Assuming the protection circuits work, there should be no damage. Only you can make the call.