Obviously it's OK to use low damping factor amps even though there is frequency variation. Many people love tube amps and many love Pass amps(solid state with low damping factor due to no overall feedback). A good place to see this is John Atkinson's graphs of amp frequency response into a sample reactive speaker load(and every speaker is reactive, not resistive like the load used to test amplifier frequency response). As a sort of rule of thumb a damping factor of 20 yields variations that can be measured but are very small. But tube amps often have damping factors of as low as 2 to 8(fairly high for a tube amp).
Of course amps have other characteristics than frequency response which is why some like low damping factor amps. Also note that damping factor varies with frequency and will be lowest at say 1 kHz. And it will be higher at frequency extremes, sometimes significantly. You need to know what frequency it's measured at. It may be OK at 1 kHz but high enough in the bass to react with bass impedance peaks.

