Feel free to talk me off the ledge

Ah the rabbit hole, most of us are in it at some point. 
I was an audiophile in the late 70’s and 80’s, then life got busy and eventually I put my dahlquist DQ-10’s in a closet because they were “too big” per the wife. 
forward to present day setting up a listening room (that was my sons bedroom!), and I broke out the DQ-10’s, still sounding amazing. 
but I got to thinking about options and a smaller footprint.  
made the mistake of listening to a pair of Martin Logan ElectroMotion ESL’s (hybrid electrostatics), and it was all over!  And down the rabbit hole I dove…simply amazing sound o can’t get enough of!

About speakers being most important: it depends. 

For years I upgraded electronics while keeping the same speakers. When I finally got some loose change, upgraded the speakers by an order of magnitude and at last discovered how just good my amp and preamp had really been all along.

The old electronics, however, couldn't handle post-CD digital. So when these were traded for contemporary equipment, the tables were turned and the speakers again became the weak link -- I was right back where I had started, albeit on a higher plane.

Now, $12k later, everything is in balance and to date I'm resisting the urge to upgrade further. Powerful temptations aside, for me the present is the logical time to pull the escape cord from the rabbit hole. Famous last words, I know.

OP,

Just a thought… “fatiguing” doesn’t usually come from speakers, although it can exacerbate it. It usually comes from the electronics in the form of high frequency hash and distortion. It takes a lot of experience to really put your finger on it… but fatigue is the sign. Your Rogue equipment is not known for it… so, I would look at your source. If it is supplying the high frequency problem then it’s going to come out the speakers. If your room is not heavily sonically dampened then that will make it worst.

Can you borrow a real high quality streamer or CD Player . Try some dampening material in the walls behind… carpet in front of speakers. I would verify this is not you problem before proceeding.

@kclone 

"Probably what, 75% to 85% of recordings are not "audiophile".  So that leaves us small percentage of stuff that truly sounds out this world on our systems."

This is the great (mostly unspoken) problem with high performance audio.

It also explains the often rather strange musical choices of some of our veteran reviewers.

As our equipment gets below a sufficiently low threshold of distortion, the realisation that the music sounds progressively worse becomes an impossible one to ignore.

@skyscraper 

"If you’re unmarried, find a wife quickly. She will undoubtedly end your developing addiction in short order once she hears about what this stuff costs."

I have an understanding and tolerant wife, but yes there are real world limits.

At least for most of us.

In any case, nowadays she's not the problem

It's actually getting harder and harder to justify buying yet more equipment to myself.

So anything I buy in future needs to be substantially better or more desirable than what I already have.

I’m with the others that aren’t fans of the Heresy.  I am a fan of the Paradigm Founders the OP mentioned