Build my own speakers?

Go for it. I started out slow by building a subwoofer first. Despite the fact that a sub is a simple box/amp/driver, there is a whole lot more to it than slapping those components together. I learned a ton. I met many great contacts as well. By the end of the design phase I was using subwoofer computer models for calculating the box dimensions to achieve a particular "Q" rating that I wanted to achieve, pretty cool stuff.

Later, I wanted to try out a single driver speaker (the ones in my System now). After reading a TON of info on the WEB, I decided to build a pair and settled on a proven design complete with detailed plans. I learned a great deal about resonance, porting, soldering as well as the many different types of resistors, coils and capacitors that are available. Here too, I met many new folks some of who have become good friends. Even if the outcome would have not been to my liking, I would have come out ahead with the knowledge I have gained and the friendships I have made.

Several weeks ago I was plotting the frequency response of my system on a test tone CD and a SPL meter. If you would have told me I would be doing that a year ago, I would have thought you were nuts!

Considerations:
1. Try out a relatively simple design first.
2. Try out a proven design, don't go to the self design route first, you will be amazed the things you have taken for grated.
3. Undertsand your wood working limitations (tools and ability) don't exceed these.
4. Have patience, don't be dissapointed at your mistakes, learn from them.
5. Take the time to meet and ask people on this site and others for help. You will be amazed at what you don't know.

Granted, my single drivers are much simpler a design than multi drivers are, but start something and enjoy the ride.

I have a couple of Madisound Odin speakers built from kits which are the surround channels of my system. They are IDENTICAL to speakers that I saw reviewed in Stereophile a while ago, at less than half the price, even with the premium crossover option.
Check out www.madisound.com

Here is another choice. These guys will facilitate a variety of DIY speaker design/build projects. I agree with the others here, you should let an experienced professional create the design, and match drivers to the crossovers. However, you could first decide what type of sound you are looking for, how much power you have, how big you want the speakers to be, and other factors, then find the right DIY project for you.
http://www.meniscusaudio.com/speakercomponents/

Do you want $20K speakers for the fraction of the price? Check these: http://www.northcreekmusic.com/#LoudspeakerKits

Thanks for all the replies. I am a serious audiophile. I have 2 degree's one in advanced mathematic functions (electronics) and the other in mechanical. The speakers i en-vie are Vandersteen 5a's. Richard did his homework as in I do. Richard also is coming out with a new speaker. I did my homework as well. I am going to design and manufacture the best speaker in the world for under $20,000. I just hope Richard will be my friend.