Kijanki - I am getting a phd in Class D circuits and have witnessed variations with temperature in PWM waveforms first hand. Whether it is audible or not is a good question but the feeback loop really has to work to keep the duty cycle in check (mainly because the voltage feedback loop is very fast and few audio modules use the slower current control loop). Tracking a musical signal is tough work and beat frequencies are all over the place, some of which are thermal.
Also, depending on the level of switching frequency, it can take longer than expected to reach thermal equilibrium. It depends on the modulation scheme and signal as to what the duty cycle is but typically, it is 50%. In this case, I doubt that they will reach equilibrium faster than Class A. Those Gryphon amps would burn your hand in less than 5 minutes despite massive heat sinks, if they weren't playing music, whereas I have had thermocouples attached to switching devices and watched their temperature rise in real time and it took about 15 minutes for the curves to level off with an output current of 100A (multiphase buck). If the devices are attached to heat sinks in addition to the multi-layer PCB, it will take much longer still (lots of mass, little energy). You can look at a Class A amp as having 100% duty ratio and 10x the bias current. It will be hard for a Class D amp to beat that kind of energy level, regardless of the mass involved.
As a result of all this, I would leave a switching amp on all the time. Besides, it hardly uses any power.
MOSFETs are actually not as sensitive to temperature as BJTs. Voltage drive doesn't pay much attention to variations in circuit impedance - unlike current drive. Not to mention having positive temperature coefficients.
Just details but thought you might be interested.
Arthur