I would personally argue that speakers are the most important (if you don't like your speakers, you won't like anything you are hearing), followed very closely by analog source. After that, amplification, then digital source. Everything else, though not unimportant, is much less important than these major components. So to the original poster's question, I would say the amp needs to be more carefully chosen than the CD player, particularly if there are other sources, especially vinyl, involved.
Most Important?
After speakers where do you focus the lion share of your funds. I known garbage in garbage out. Would you buy a really great amp and an ok cd player of vice versa?