OK, here's my detailed description of what jitter "sounds like." Really, the question is how jitter impacts the music, ie. how it negatively impacts the music.
Jitter will destroy the microdynamics in the music that allow you to "enter in" to the music and enjoy it. Details will be "blurry" and thus imperceptible or require unusually high amounts of concentration to perceive, since they are experiencing artificial time distortion and thus your brain has a hard time recognizing them. Phase gets messed up and the details can often literally disappear. This tends to cause macrodynamics to be emphasized, so you will hear all the "big" sounds just fine, but they will be artificially pronounced because all the little details are smeared away.
This makes for boring, not moving, music.
Jitter *can* also have impact on a particular frequency range of the music. Often, it will result in diminished bass. Musicians will tend to "wobble" or "flutter" L R rather than be fixed like a stone in a specific place. Instuments will seem to come from one or the other speaker rather than from somewhere between the speakers.
When you listen to a low-jitter recording on low-jitter equipment, you will have the experience of hearing the entire musical landscape in wonderful focus and you will be able to see deeper into the music, rather than experiencing a flat, macro, uninvolving presentation. Music with a lot of jitter won't generally offend you tonally (like "too bright", etc.) but it will simply not present the detail you need to stay interested for very long.
If you had a switch with which you could turn a certain amount of jitter on and off, you could switch it back and forth and you would probably *NOT* notice an immediate sonic difference. However, with the jitter off, you would find that in 30 seconds you would be enjoying the music and experiencing it rather than trying to follow it by concentrating. Jitter will impact your ability to enjoy the *rhythm* of music. You can definitely notice the difference, but it is not a tonal difference that you will pick up immediately. It is a "bluriness" difference. It is a "how much do I have to concentrate to really experience the details of this recording" difference.
Also, remember that jitter gets worse when you have bad power being delivered to your system. So if you add a good power filter to your system, or a balanced power unit, and you get more focus, detail, musicality, etc., a lot of that probably has to do with jitter reduction caused by supplying clean power to your equipment. Power fluctuations will introduce jitter during playback. If your system sounds more focused and musical at night, that may have a lot to do with your AC power being cleaner and your digital gear giving you less jitter.
You can also reduce jitter by ripping your CDs to a PC and using a pro audio card to send the data to your DAC. This eliminates transport jitter, which is jitter caused by imperfections in the mastering of CDs and imperfections in the spinning and reading of the data in realtime (1x speed) by the laser.
Also, remember that many CDs will be mastered with jittery equipment, so the data you have on CD is un-dejitterable. But you will find that some CDs have low jitter and will come into focus much more when you clean up your own system's jitter problems. Others won't.
Hope that helps :)