The stepup transformer should absolutely be used with the moving magnet setting on the Vincent, not the moving coil setting. Using the moving coil setting will cause the cartridge to see a load impedance of a fraction of 1 ohm, which is the reason for the low volume you experienced. Paradoxically, it will also result in too much overall gain.
That is all explained in this article
, which is lengthy but provides a lot of good information.
Also, I suspect that it would be within reason to try the 30 ohm position of the FRT-3, and choose whichever of the two positions sounds better. Trying that position won't do any harm, although I would not change the setting while the system is powered up, or at least without first turning the volume control all the way down.
You'll find yourself turning the volume control up a little higher in the 30 position, btw, which might slightly increase background hiss.
It might also be worth trying the Vincent's moving coil setting without the transformer, or with the transformer in pass-through mode.