The HiFi market (sorry Audiophile) market is so small nowadays, the manufacturers are small, they're generally a handful of guys who are putting this stuff together, there's one guy who started the company who knows a bit about circuits. Their payroll is probably around $200K/month, and the owner needs to make a profit, and when you consider that you may only sell 10-20 units a month, you can see where the price comes from.
If you look in the mass-produced hifi gear from the 60's and 70's, those were relative bargains compared to today since they were selling volume. HiFi was mainstream, and there was a lot of competition. Look inside one some time. They're built like tanks, transformers that weigh more than a small child, and many of them are still working 40-50 years later.
So no, the gearis overpriced from the standpoint of looking at the materials involved but you've got to face it that this is such a niche market anymore that in some ways it's amazing there are as many companies as there are.