Dirt on my stylus.....what to do?

I've noticed that I have some dirt and dust (and probably more junk that I can't see)on my stylus. I have tried my stylus brush with a bit of LAST STYLUS CLEANER on the brush (I've been told not to put LAST directly on my stylus) and the dust is still there.
What do you recommend???