The term 'audiophile' means 'audio lover'. So goof off with it to your heart's desire :)
A good number of tone controls, when set to flat, are not in fact flat at all- there is usually some perturbation on account of the controls. In addition, an additional stage of gain is required to drive the tone control circuit. Unless switched out (and even the switching circuit can have an audible artifact) these things will contribute to a loss of bandwidth, detail and increased distortion.
Hence, many 'purist' 'audio lovers' do without, in the quest for greater transparency/immediacy to the music. So, to refine the definition, there is a difference between an audiophile and a purist (purist being a subset of audiophile).
But I have to ask- because I don't know- if you have hearing loss (like many of us over 40/45 years) how is it that tone controls will help? Won't that make the experience of the recorded music that less real (because you could never hear it that way live)? Or is that not important?