Airborne oil, moisture, and dust slowly polymerize into a sticky yellow film on surfaces, bonding especially well to glossy finishes. Dust can also slip through ventilation gaps on the tops of components and accumulate inside, where it becomes difficult to clean. Buildup on speaker cones and drivers is far harder to remove and can be detrimental, as it alters the vibration characteristics of the drivers and negatively affects sound reproduction.
I certainly understand the visual appeal from an owner’s perspective. Cover them before going to bed, when you can’t see or appreciate them, and uncover them during the day to enjoy their appearance. If you have guests, by all means uncover them for showing off and visual enjoyment.

