i do not understand. your av processer has 3 channel outs not 5? what is decoding your dolby 5.1 and breaking the signals into 5 seperate outs? what about the sub out .1?you might want to name the equipment and someone can help you further.
I have posted several times on this subject and use my two channel and HT system together, exactly the way you describe.
By returning my audiophile preamp to the same volume setting before each movie, my Dolby 5.1 processor controls my front (stereo speakers) at exactly the same ratio and volume as the HT speakers. My stereo system is playing "inside" the HT system, and it is possible to make them integrate successfully, while not damaging the quality of the stereo when the HT is off.
Best of all, the cables to the dedicated stereo system need never be unplugged or moved about. There is probably a superior way to do HT, but if two channel is your passion (as is mine), this is the only way for the two to survive together without harm to the dedicated music system.