I guess it all depends on what you normally listen to and how you listen.
What - do you listen to a lot of surround music or just surround movies, or stereo music
How - do you use Pure Direct mode or do you engage Audyssey EQ?
If you engage any bass management or Audyssey, you are digitising the Oppo's analog output and then performing DSP on the signal, before using the Marantz's DAC anyway.
A. If you use the system for surround movies and stereo music primarily, here's what I'd recommend:
Stereo preamp with XLR inputs for stereo listening. Oppo's XLR stereo outputs are hooked directly to the stereo preamp. The Marantz's stereo pre out goes to the preamp's HT bypass input.
The AV8801 is a decent preamp but it won't IMHO outperform a reasonably good stereo preamp like my old Bel Canto Pre3.
For movies, use HDMI from the Oppo and engage Audyssey's room EQ.
If you have connected both 5.1 RCA and HDMI outputs from the 105 to the 7005, both are active and either should be selectable by switching among the 7005 inputs. When you select the 5.1 RCA inputs, you are using the DACs in the 105; when you select the HDMI input, you are using the DACs of the 7005. There is no advantage to using balanced cables unless the 105 is a very long distance from the 7005. Consult the 105 manual about using the stereo outputs for the front LR channels of 5.1 so you take advantage of the superior DACs.
You should be able to compare the sound with the 105 DACs to that with the 7005 DACs just by switching among the 7005 inputs. Be aware that there is a bug in the speaker distance settings of the 105, so it's easiest to start by setting all the distances to 0. A fix is said to be on its way soon, and if you have the 105 connected to the internet as it should be and you set the menu to update firmware, you'll get a notice when new firmware is available.
Finally, if you don't have analog sources, you might consider using the 105 direct to your amps and selling the 7005. At least you should try bypassing the 7005. The 105 can provide speaker configuration, bass management, and fine volume control.
IME, there is definitely a reason to use the 105's balanced outputs even for short runs. At least in my system. The balanced outputs provide much higher gain and keep the gain level in the same ballpark as when playing other sources (vinyl, open reel tape in my case). Using the unbalanced (RCA) outputs requires setting the volume level on my Aesthetix Calypso linestage much higher. Yes, I could run the 105 directly to my power amps but the sonic improvement using the Calypso is not subtle. I have the Modwright version of the 105 but the same thing happened before I got it modded.
According to Oppo there is no sonic difference between the single ended (RCA) and balanced (XLR) outputs except for the level of the XLR being 3 dB higher. If that 3 dB is important for the following equipment, there is no reason not to use XLR if connections are compatible. If Dopoque is using analog from the 105 to the line stage, then the line stage must be coloring the sound in a way he prefers; if he is using a digital connection, then he must prefer the analog processing of his line stage to that of the 105.
Morty74 might want to connect the output of his cable, satellite, or TV to the HDMI input on the rear of the 105 so that source can benefit from the superior DACs of the 105, and use the input selection button on the 105 remote.
Oh, yes, I much preferred colored sound to the bleached, threadbare "transparency" I get from going direct or via a passive linestage.
A bit snarky, Dopogue. I was just pointing out two explanations for your preference, not criticizing it. Do you really perceive the sound from an unmodded 105 as bleak and threadbare?
Sorry for the snark. I didn't bother trying the stock 105, before
sending it to Dan Wright, in direct/passive mode. I had done this
(several attempts with different passive linestages) with my BDP-83SE
and found it sounded exactly that way. I figured the 105 wouldn't be
that different, but then I've never found a CD player that didn't benefit
from an active linestage. They probably exist. I just haven't heard
BTW, the Modwright mod is well worth the cost, to me at least, which
is twice what the stock 105 cost.
EDIT; A friend brought over his stock 105 last weekend and we
Dopogue, I think I prefer the sound of my 105 through my Parasound JC-2, but it's a close call. The JC-2 seems to take just a bit of edge off the 105, making it sound smoother and more elegant. But, of course, that means the JC-2 colors the sound in a way I prefer. My other sources are a Sony XA5400ES and a JC-3 phono stage (Thorens TD 124, SME Series III, Ortofon SME 30H).
I switched from balanced to single ended because I use unbalanced KEF KUBE's with my 107/2 mains and 102 surround. The Jensen ISO-MAX transformers I used to go through the XLR-SE transition added too much clutter, given that noise remained inaudible with the SE cables I got from LA Pro Audio.
I second Dbphd's suggestion that you sell your Marantz 7705 and run all Oppo outputs direct to your Rotel. I found I liked the sound of my system better, for both ht and 2-ch music, once I eliminated both my Parasound AV2500 preamp/processor and my VTL 2-ch preamp. I use the 'Dedicated Stereo' xlr outputs to go direct to my front L+R front channel amp and the rca surround outputs direct to my surround amps.
If you're unsure, just try your system with and without the 7705 and let your ears decide. The direct method will be very neutral and without coloration which may or may not be to your liking.
The only reason you may not want to use this method is if you have multiple sources since the Oppo has very limited inputs.
Typically in audio the simplest, shortest and most direct signal chain is preferred and will tend to sound the best.
There is a flaw in the observation made in my 4-26-14 post. The output of the JC-2 preamp goes through the KEF KUBE, whereas direct from the 105 does not. The KEF Reference 107/2s were designed to be used with the KUBE. My observation about taking the edge off may be more likely attributed to KUBE/no KUBE rather that preamp/direct.
If you read John Atkinsons report in music in the round .He reviewed and tested the Marantz AV 7005 and found that the signal to noise ratio was about 5 db better using RCA cables in lieu of XLRs . I have the 7005 and found that by switching from xor cables to rya cables gave me at least 3 db more volume.