Do you swap cables when using a headphone amp?

My phono stage has one output which will be connected to a Pathos Classic One MKIII. The Pathos does not have a headphone jack so I want to get a great sounding headphone amp to use with my Sennhieser HD650s.

I've never used a headphone amp and I'm assuming you would connect the phono stage output to the headphone amp input. But it doesn't make sense that you would swap the cable every time you changed from headphone to speakers or visa versa. What am I missing here?
If your Pathos has a tape out, you can connect the headphone amp to it. If it doesn't, you will need to use a Y-connector or swap cable.
or a switch box or pass the signal through the headphone amp to the Pathos, if you get one that has that capability.
Excellent! Thank you. The Pathos has Tape in and out loop. It's been 20 plus years since I've used a tape recorder that I just looked at those connections and thought, "Well, I'll not be using those for awhile."

I just thought that with this nice gear, there had to be an elegant solution.

I would also think that the preferable way would be to come straight out of the phono stage into the headphone amp.
Tape loop is the way to go, unless you buy a headphone amp with a loop out on it. Either way, should not make much of a difference either way.