Sometimes the holes are big enough but there can be some loose "swarf" present that needs to be dislodged.
If the hole genuinely is too small I use a small penknife with a straight sharp-edged blade to ream out the hole. It only takes a few seconds and you only need to skim it. The "scrapings" are usually statically charged and tend to stick to whatever they touch so every trace should be carefully removed before playing (being careful not to drop them on the grooves) so that you have nice clean surfaces. I don't think that loose core on the grooves will damage them other than causing mechanical interference at the interface of LP and a solid platter/mat. My only concern would be that the stylus might react when it hits the debris.
FWIW, I think the problem is also related to the turntable's spindle tolerance as well. I've noticed that discs which fitted easily on some turntables required to be reamed out on others.
There is also the issue of mis-registered holes but that might be a meaty subject for another thread ;^)