driving headphones with preamp direct RCA output

Okay so I have heard of this being done, what does it take?

If you have a standard preamp with RCA outs, it will drop somewhere between 1 volt and 4 volts lets say to the amps, it has an output impedance of like 1.5 kohm would it drive a very high impedance headphone at 250 ohm, sensitivity of 98 db with reasonable power?

If so I see no need for a headphone amp right? I have not tried it, but I did with a preamp a long time ago and some cheap headphones which were probably just way too low of an impedance of like 32 ohms or something and it did not work.

Thanks, I don't own anything for this, but was considering trying it and buying a pair of highly efficient high impedance headphones to give it a shot if it even could work.
I have not used the SE outputs but have listened to Balanced HD650, Sony R10, and Sony Qualias through the balanced outputs of the preamp I am using now. I have also used the HD650s through the balanced outputs of my Audio Aero Prima SE DAC which has volume control. I would think a 1/4" to pair of RCA adapter would work or just have your headphone cable reterminated.