Why have capacitors improved so much over the years?

Not just capacitors. Chips, transistors, resistors... almost all electronic components had been improved over the years.
Btw, in terms of capacitors, electrolytic capacitors been improved greatly.

I can't understand either why manufacturers go through all the trouble of designing a piece of gear & then at the most critical point insert cheap caps.

When design engineers select components/parts for their circuitry, it is their responsibility to assure all components/parts are up to required specification and not to compromise safety and reliability.
Most boutique audiophile capacitor manufacturer cannot provide required documentation for design engineers to consider installing such capacitors in the design.