Eliminate USB from the chain?

I borrowed a Naim UnityQute2 over the weekend that I ended up really enjoying with my Harbeths. I get the Naim/Harbeth love now, even with an entry level unit no longer in production.
One thing I noticed when A/B’ing with my setup is that things sounded cleaner and clearer through the Naim at a microdynamic level, esp at low volumes when streaming Tidal. My SMS-200 into Ideon Audio 3R v.2 (makes a huge difference for the better IMO) into the Job INT amp sounds impressively clean and spacious, for a USB setup but I feel like the UnityQute2, with just ethernet going straight in and no USB in the chain had some sonic advantages that brought it closer in some ways to CD or CD rips on a server. Anyone else have this experience? Makes me rethink everything really. 
We buy all these high end cables and reclockers etc to improve the USB signal when just eliminating USB entirely from the chain seems, in my limited experience admittedly, to be the simplest and best performing option. I'm now looking at all in one options as potentially superior for those doing a lot of streaming. would a higher end USB streamer/server get me closer? I'm sure but then we are throwing more and more money at it just to reach the level of an entry all in one.  While my USB setup is entry level, so is the Unity so it's a pretty fair comparison I believe. 
Anyways, I know this is just small sample size so I'm not trying to draw any definitive conclusions personally from this. So I defer to others here to see if they have had similar experiences when moving to an all in one unit that eliminated USB from the chain completely. Cheers!