Using TV as Audio Hub?

OK - That title probably makes no sense. Here's the situation. In our guest room/playroom/gameroom, we have a 37" LG LED TV, with the following sources: DirecTV receiver, Sony BDP, Apple TV and Wii. I want to hook up a pair of PSB bookshelf speakers I have lying around for a simple 2-channel system.

I want to get a simple receiver or all-in-one type unit to run the sound to the speakers.

Since all of the sources are already hooked up to the TV set, and the TV has an optical output for audio, I could just run a Toslink cable from the TV to a DAC and small integrated amp (or a combo DAC/Integrated), and let the TV handle the source switching.

If I do this, will the sound quality be materially worse than if I ran all of the sources directly to the receiver for audio?

I was considering getting something like the Wadia 151 PowerDAC ($800) or an NAD VISO TWO ($500).

Thoughts? Simple is best.
I don't have any experience that would help with your choice of the TV as the center of the system but another all-in-one unit to consider is the Onkyo A-5VL integrated with built-in Burr Brown 192 kHz/24-bit DAC. It's very nicely built, slim unit that sounds great and is a real bargain since the price has been cut to $350 from $700.
if you're not doing a lot of critical listening in your guestroom, i'd go to accessories4less, vann's or comparable and get an avr--you can find a serviceable onkyo, marantz, etc. for less than $300--then connect all your audio and video sources to the receiver, with a single hdmi cable to your tv; obviously you can go used and spend even less.
Thanks for the tips. I'll check out the Onkyo. Some people have described it as a bit shrill, but the price is outstanding.

As for the AVR idea, that was my first thought, but I'd rather not go that direction because (a) I want something small in size, (b) I only need 2 channels of amplification, (c) it would be simpler for my kids to operate if the TV did all the source switching and they only had to use the amp to change the volume, and (d) I find the HDMI source switching in my main rig to be a bit glitchy (I have an Anthem AVM-50v).

I suppose I could also get a very small integrated amp and small DAC, like the MF V-DAC, and run the optical out of the TV. I'm mostly interested to know if the TV will degrade the sound materially.
Consider running a key or several pieces from the tv and also directly to your new Peachtree. That way, you have the convenience of everyone using the stereo controlled by the tv and when you feel critical, you can avoid the tv connection muddling things up.
I am positive that my tv's digital out is in no way as good as running source direct to my system.
Elevick - that is a good idea. The only problem is that I need a DAC or receiver with 2 Toslink inputs, which eliminates all of the ones that have been proposed so far. I need to go back to the drawing board, I guess.
For the life of me I don't know why everyone has to make things so difficult.The system you have in mind will work perfectly so just stick to your original plan & run an Toslink from the tv to a nice little all in 1 like the Wadia or a Peachtree Audio & enjoy.
I have a Peachtree Audio MusicBox with Sonus Faber Toy Monitors for PC based audio,I also run a Wireworld Glass F/O Cable from TV to the MusicBox for video & it sounds AWESOME!
This is exactly what I do in my family room with DirectTV, APpleTV, and Blu Ray running to TV , then out to a Bel Canto Integrated w/ DAC via TOSLINK. Works pretty damned well. I am pretty sure my DAC is reclocking the bits and bytes so unless there is loss somewhere in the path, it should be identical to direct from source.
I'm doing the same. HDMI from all sources into the TV and digital out into Wadia. Indistinguishable from CDP direct digital, or laptop USB into said Wadia. The convenience and simplicity cannot be beat - especially with nothing lost in translation (the TV does only pass-through, no resampling as best as i can tell). YMMV.