With a DC motor, the speed is set by voltage level. Small changes in voltage can affect the speed.
It is possible that maybe in the switch(or something) that very small variations in the voltage passed to the motor are happening as you turn the PSU off and on. This could cause a very minor speed change, and account for what is happening.
This is just a guess, because I can't be there to actually test and diagnose, but it could be that.
This is why some DC motor TT systems(like Teres) use a strobe on the platter and an automatic motor controller. Additionally, a non-controlled DC motor can slightly slow down during the course of playing one side of a record, due to stylus drag.