"naked" listeners

I recently moved and just for grins pried off my speaker grills to listen for any improvement. It worked! But now I wonder. How do you "naked" listeners cope with dust build-up on your drivers?
A lot of speakers sound better with the grilles off, but Avalons(what I run) use the grilles as an integral part of the system, yea they sound better with the grilles intact, sort of weird. I am sure a feather/static duster would do the trick for keeping your drivers clean.
My theory is dust will not settle on vibrating diaphragms, so put the grills back on when you are not listening. I am genius, no?
Swifer *charged* cloths. I clean my speaker cabinets, cones, and "gear" with them. Though technicaly I use the Endust badged product, since I can get it a little cheaper in my area.
I couldn't have said it better than Blkadr above,and if you leave the grills off over night and dust gets on them I would think as soon as you start to play them the dust will get blown off and settle on the floor.
A well designed speaker is compensated for the loss in high end due to the speaker grill cloth. The reason that taking the grills off makes it SEEM better sounding is that it's a tab bit brighter. A few more dB's in the high end. Unfortunately, like BLUE TV's in the Best Buys of the world, it sells more, but not truly more accurate.

If you like brighter, GO FOR IT. I prefer audio as a pursuit of the perfect reproduction of the live, hence I keep them on...
If you look at the grillframes and the well finished baffles of many speakers you can infer that they were meant to be removed for serious listening. Some manufacturers take great care in designing the grillframe and intend the user to leave them on. It depends on the speaker you have.