You need a real phono preamp. Any phono preamplifier can be plugged into "line in" inputs. The rm-4 is apparently a pre-pre phono; it only brings the signal level from a moving coil cartridge up to the level that it can be used by a phono preamp. You should say what cartridge you're using for a more explicit answer. If it is a moving magnet type cartridge, you would not need the rm-4 at all.
No PHONO IN Solution
The headphone amplifier I use doesn't have PHONO-IN input.
I always thought that MC/MM phono preamp such as Music Reference RM-4 would be sufficient. But as it turned to be the device like this only brings the signal up to the level when it can be fed into the PHONO IN input of the amplifier that does already has a PHONO-IN jack. I wonder what I supposed to do if the amplifier doesn't have PHONO IN build in. What category of devices can be used?
I always thought that MC/MM phono preamp such as Music Reference RM-4 would be sufficient. But as it turned to be the device like this only brings the signal up to the level when it can be fed into the PHONO IN input of the amplifier that does already has a PHONO-IN jack. I wonder what I supposed to do if the amplifier doesn't have PHONO IN build in. What category of devices can be used?