About the interesting question of the Op...
A very interesting question as usual from an artist...
I remember the moment where i really became an "audiophile" ...
The spark was set, the seed sowed when i was young ...
But i listened music in a state of complete ignorance about sound...
Why ?
Because i had no idea what to buy...
I had no idea what to do to improve what costlier gear i must or may buy, worst i thought that buying costlier gear defined what was an "audiophile"...
I was interested by music only because music matter the more for sure but by ignorance also even if i dreamed about a high end system, i could not afford it anyway...
Audiophile was synonymous of big money...
I retired from my job, i had plenty of time then with a dedicated room, a low cost system, i begun non stop experiments in acoustics to improve my system...
I became an "audiophile" and realized it has nothing to do with money, but about acoustics knowledge ( Not just room acoustic by the way )
Now i am in ecstasy with my low cost speakers and headphones...
All modified and optimized...
Am i ashamed to be an "audiophile" ?
No, because it is synonymus with acoustic learning nothing else in my mind...

The label stick pejoratively among people who did not know what to do seeing those who are able to afford anything and did not need to experiment and learn so much...