Thats a big question. When I first read your question I didnt bother because there is so much to say. I'll add my 2cents to hte good stuff above. (Now its filling time during a short bout of insomnia.) Somebody could ( and has) write a small book about it. Ive included a number of sites that I think are good because its easier than reinventing the wheel.
1) One of the better online glossaries of tech terms is at Aikens site. It includes bypass and coupling caps and is often helpful if you are looking for tech answers (at least in tube circuits). http://www.aikenamps.com/AmpTerms.html
2) A short article about bypass caps. http://www.seattlerobotics.org/encoder/jun97/basics.html
It defines them as follows:
Bypass capacitor: A capacitor employed to conduct an alternating current around a component or group of components. Often the AC is removed from an AC/DC mixture, the DC being free to pass through the bypassed component.
This is just what you want in a power supply. You want dc current. The ac is rectified to lumpy dc and then caps are used for more filtering and/or bypassing the remaining ac to ground away from where you are going to need the dc.
When you think about using caps in this way it helps to remember that caps in parallel add up. This just the opposite of resistors and inductors. Read Adding components in series and parallel here: http://www.aikenamps.com/TI_Aiken_int.htm
3) You probably already understand that it is the nature of caps to allow higher Frequecy to flow and to block lower F. Therefore, if you put a cap directly in front of something (another stage or a driver), that is in series, it will block the flow of lows from getting there. This is how coupling caps are used. On the other hand, if you make a separate parallel path (like a fork in the road) in the circuit and put a cap in it, (that is use it to bypass) the High F will take that route and the lows (including direct current which you can think of as infinitely low frequency) will go the other way. This is how caps in power supplies and tube bias circuits are used. Here is a good site on the basics of capacitors. http://my.execpc.com/~endlr/index.html
You might wonder. Why not use just one cap. Why send the current through 2 or three caps in parallel? One reason: In theory all caps become inductive as frequency rises.(that's right, caps start to act like inductors). You want the caps to be a perfect short at all frequencies. Inductance, of course, increases with frequency. If they are paralleled they can more resemble a perfect ideal cap (less inductance) and deal with high frequencies better. There are other reasons too. This has become almost a religious topic among diyers. Many folk mix and match many combinations in various circuits and report better/worse sound and Im not certain there is anything definite you can point to. There are certainly differences in caps that would lead to different performance and people mix and match different types and sizes. If you read the article directly above it gives you an idea of different designs in caps. Here is the old famous article that started lots of folks looking to tweak sound with different caps. Young & Marsh, Picking Capacitors: http://www.capacitors.com/pickcap/pickcap.htm
4) Above by GS >>Actually, "coupling" is a proper description. The capacitor couples two circuits together by the common capacitive reactance of both circuits.
I agree here for the most part. I think coupling is a helpful and accurate description of what is going on. You are transferring power from one stage to another. It is very helpful to keep that in mind and coupling gives that a name. I think I would use impedance instead of common capacitive reactance though because impedance includes the resistances of both circuits and the R is a factor too. (the time constant of the charging and discharging circuits, for example, is R times C) Maybe I just misunderstand GS as people use different terms.
5) Bypass caps are used in various places and have different functions. People spend a lot of time talking about bypass caps in power supplies. There are other kinds of bypass caps. As noted above by GS and Aball. If you have tubes most voltage gain circuits, for example, have a bypass cap in the bias circuit of the tube ( from cathode to ground). It decreases the gain and keeps the bias voltage at a more constant level. It more or less provides a path (bypass) for the ac signal to ground. It gives the signal a different path than through the cathode biasing resistor. (If the signal current goes through the bias resistor it will cause a voltage drop in the resistor. This voltage change will cause the bias of the amp to change because the bias is set by this resistor). If you want a visual and a little more on this look at the second picture here. The cap labeled Ck is a bypass cap. It's purpose is explained in the texthttp://www.tpub.com/neets/book6/20h.htm
I tried to find a nice picture of caps used in a Power supply but I think Im getting to tired.