It's worth experimenting with this, but I'll bet anything you'll be missing some midbass and low end. In this situation, you're essentially running your phono stage as a passive preamp. Even if you ideally match the output impedance of your phono stage to the input impedance of your amp, I bet you'll lose the fullness that a linestage brings to the picture. I've had several excellent passives and this has always been the case. Ralph Karsten of Atma-Sphere fame has an excellent explanation for this phenomena in this forum:
Are passive preamps better?
Are passive preamps better?