Probably the amp. I don't see how they get 25 watts out of EL84's. You'd be lucky to get 18. If the stock tubes are still in, that's another source of poor sound. Tube rolling may help a little but a bigger amp will help more.
speaker or amp distorting
i have jm reynaud arpeggione floorstanders that are a small two way with soft dome tweeter and 6.5 inch paper cone in a transmission line design. they are reasonabley efficient 90db. my amp is a modified jolida 102b with el84 tubes and 25 watts of power. on simple music with without too many dynamic peaks and modest instrumentation i.e. four or five instruments or less this setup is magical. however, with big band or vocals when things ramp up dynamically the treble seems to harden and hurt my ears. occasionally i will hear some distortion. is this the amp or the speaker or something else? thanks