I have used an Aleph 5 for awhile now, after having an Aleph 3 for awhile on a loan before that. I find that the 5 has more controlled bass, and an overall authority that controls my speakers in a cleaner fashion than the Aleph 3. However, both are pretty similar in their midrange lushness and the very smooth top end. I found both amps to be quite listenable and not fatiguing for extended periods of listening.
I know from chatting with Pass Labs that they do believe that there is a general progression in peformance climbing the ladder. I would hope this to be true, as there's got to be something other than power rating to justify price increases in products? However, the current source and internal components are all quite similar, and therefore, the two models probably are pretty close in a lot of respects. I do not think you'll be unhappy if you get the 5, though.