1. You can't use a single tone to match levels because of its interactions with room modes that make some frequencies significantly louder or quieter than others depending on location (even 6" can make a huge difference).
You want to use pink noise containing bass frequencies with the same bandwidth within main speaker and sub-woofer pass bands (for example 1 octave from 60-120Hz) and match the level between a speaker and the sub-woofer with one channel input or both together (the pink noise must be in-phase).
2. RELs are relatively simple to use well with the sealed stand mounted monitors favored by audiophiles. Sealed enclosures have a second order roll-off which will sum-flat with the REL's second order low-pass and excursion remains constant below their pass-band so IM distortion is less of an issue. With a 60-80 Hz F3 point placing them so the SBIR null is outside their pass-band is also easy.
That's not your situation and using the REL's built-in cross-over instead of a separate sub-woofer controller with a higher cross-over point and high-pass filter on the mains (ideally iwth an independent frequency adjustment) is sub-optimal. You're doing nothing to reduce IM distortion inducing excursion in the speaker which will increase to what it would be without an enclosure and are going to have issues getting flat response matching the speaker's 4th order high-pass response with polarity inversion in the port's pass-band with the sub-woofer's low-pass.
Part of your bass problem is a big notch at 200-300Hz which results from the main speaker's SBIR null and nothing you do in the last octave is going to fix that (you need to bring the speakers farther out; preferably with 4-5' from their front baffle to the wall which will put the SBIR null at 70Hz or below and out of their pass-band with an 80Hz sub-woofer cross-over). Other issues are coming from how the speakers are coupling to the room modes which can be worked around by getting those frequencies out of the speakers and into the sub-woofer which doesn't need to be placed where you get the best high frequency performance.
You want to use pink noise containing bass frequencies with the same bandwidth within main speaker and sub-woofer pass bands (for example 1 octave from 60-120Hz) and match the level between a speaker and the sub-woofer with one channel input or both together (the pink noise must be in-phase).
2. RELs are relatively simple to use well with the sealed stand mounted monitors favored by audiophiles. Sealed enclosures have a second order roll-off which will sum-flat with the REL's second order low-pass and excursion remains constant below their pass-band so IM distortion is less of an issue. With a 60-80 Hz F3 point placing them so the SBIR null is outside their pass-band is also easy.
That's not your situation and using the REL's built-in cross-over instead of a separate sub-woofer controller with a higher cross-over point and high-pass filter on the mains (ideally iwth an independent frequency adjustment) is sub-optimal. You're doing nothing to reduce IM distortion inducing excursion in the speaker which will increase to what it would be without an enclosure and are going to have issues getting flat response matching the speaker's 4th order high-pass response with polarity inversion in the port's pass-band with the sub-woofer's low-pass.
Part of your bass problem is a big notch at 200-300Hz which results from the main speaker's SBIR null and nothing you do in the last octave is going to fix that (you need to bring the speakers farther out; preferably with 4-5' from their front baffle to the wall which will put the SBIR null at 70Hz or below and out of their pass-band with an 80Hz sub-woofer cross-over). Other issues are coming from how the speakers are coupling to the room modes which can be worked around by getting those frequencies out of the speakers and into the sub-woofer which doesn't need to be placed where you get the best high frequency performance.