Why Do Some Recordings Sound Great and Others Dead?

@OP - If you feel uncomfortable, here's my recommendation:

Corner Bass traps or soffit traps from GIK.   The soffit traps work a little better, but the corner bass traps give you more floor space.

I have them turned down so I don’t have too much of it.

This is exactly part of the syndrome!! 🤣

When you clip the peaks, the subs will sound too low and you will want to bring them up in level but now they’ll sound super smooth and natural.

This article I wrote will help you.  It is more about fixing bass problems than avoiding a sub, so if you read through the tech parts you'll know what to do.

It’s a common symptom of narrow, but tall peaks in the mid-bass or bass. Clipping them with an EQ/DSP is often the answer.

The reason it affects some recordings but not others is because the peaks are pretty narrow, so many recordings don't trigger the bass node.

While you CAN fix this with a variety of ways, like moving a sub, speakers, bass traps and moving your listening location, these particular symptoms are easy to deal with via DSP. 

Of course, other methods may give overall a better, broader solution.