XV-1S and Antiskate

The XV-1s is a fantastic tracker, so much so that changes in the anti skate don't seem to effect the sound that much. Most people (based on searches) seem to set it at zero or very low. I have been listening with no anti skate based on the advise of my dealer. I decided to put a grooveless record on and noticed the arm (Phantom II)swung in fast. Out of curiosity I increased the anti-skate until the arm didn't move. The sound changed slightly and towards a warmer sound. I am more comfortable with the grooveless record approach for setting antiskate because the wear on the stylus would be more even. Some people say set it by ear but does this make sense with a strong tracker like the XV-1s? Am I missing something?