Starting a high-end audio shop... Am I crazy?

My best advice is to keep your day job, at least at first. As a life long music lover I became a partner in an independent record store. Started out with three stores and ended up with barely one. Learned a lot, met a lot of great people, collected a lot of vinyl and cds and loss over 90% of my investment not counting no income from the store for years. Sometimes it's not a good idea to mix your hobby(passion) with your livelihood.

I am actually thinking of giving up a $120,000 yr job and starting a high-end audio retail biz - both online and hopefully later on a brick & mortar store

From a business point of view, and assuming you don;t have a revolutionary business-model in mind, I would definitely join Rec's suggestion to keep yr present job (at least for now).
Highly fragmented market;
Higher margin products are very slow-moving;
The hi-end target clientele is small -- as opposed to consumer electronics. Worse, the advertising support is minimal -- as is the corporate equity manufacturers have compared to, say, Sony.
It's difficult to create and retain a loyal client-base;
Hi-end audio's value proposition is weak (the excitement of "good sound").
Do you have a value proposition (that hi-end clients will buy into)?

With your sales & marketing background, I am sure you've studied the issue extensively. However, I believe that trying this out as a "paying hobby" before you give up everything else would be more opportune!

As always, there's also lots of merit in pursuing a dream! Cheers

given the sales of used equipment on the internet, transhipping and the consumer's goal of getting a discount when purchasing components, you are taking a big risk in retail.

in addition, there are many manufacturers who are selling direct. they are your competitors along with other dealers.

I know 2 dealers that cater to 2 channel only. One of the dealers comes from a wealthy family so whether he makes money or not doesn't matter to him. The other dealer recently told me that if it wasn't for 1 great customer he has he would close up tomorrow. I say keep your job and dabble on the side that is my advice.

People are just one mouse click away from a better price these days.
How can you compete with that?