Subwoofer calibration quandary...

I hope that I understand your question. If so, there will be many opinions out there, but since you asked......
I believe that it is best to scheive the flatest overall response with a minimum of distortion. Different areas of the room will have different peaks and valleys, so you may have to decide to optimize the response for one or two seats. It is worthwhile to try for +/- 5 dB across the spectrum, but it is difficult to acheive, even with equalizers and room treatments. I would not try to push the sub any lower than it can safely go without distortion (-3 dB). A good sound meter can help plot out your room (tedious) or you can pay for the service. Getting a sub to sound right is difficult and maddening. So much trial and error, but hopefully you will know when its right and actually have some fun trying! Good luck!

Thanks for the response. :)
Here's what I have so far according to my RS meter and Test CD.
Listening (measurement) position is at 3/5ths of both room dimensions.
First-order modes are 25 and 33hz.
The sound of the curve shown is basically weightless in the upperbass, while dinosaur foot-stomps dislodge knick-knacks.
Bass instruments have the requisite grunt, but not much personality.
I can iron out most of the upperbass valleys, but the low and mid bass end up at +9db.

db - hz - (deviation from 1khz reference; db)
73 - 20 - (-2)
74 - 25 - (-1)
74 - 31 - (-1)
74 - 40 - (-1)
77 - 50 - (+2)
76 - 63 - (+1)
66 - 80 - (-9)
72 - 100 - (-3)
64 - 125 - (-11)
71 - 160 - (-4)
72 - 200 - (-3)

In my experience, your measurements are typical. Changing one parameter changes all others. Where is your sub placed in the room and in relation to your mains? What is your crossover point? What main speakers do you use? I find that placing a sub in a corner will excite several modes, depending on where you are sitting. I moved my Hsu VTF-3 out from the corner. It has plenty of volume for my room, so i didn't need to place it in a corner. I think my overall response is smoother. I am using floorstanding Dynaudios which are rated down to 34 Hz (-3 dB) anechoic response. I only use the sub below 50 Hz, so it is easier because I have fewer frequencies to worry about. I can't change my mains' position much, but I did move my sub (80 lbs.) around quite a few times!

The sub is a Paradigm Servo-15 placed against the center of the 22' front wall, 3' behind the main speakers.
The mains are B&W CDM 9NT crossed over at 80hz @ 60º Phase.

I'm currently using a 35hz crossover @ 30º Phase, and the results are slightly better if I can accept the slight reduction in 20hz extension.
The new phase angle seems to partially mitigate the 50-60hz hump and 80hz null. (the hump begins to reappear if the level is raised much further though)

BTW, if I have one nit to pick with the Servo-15, it's that it's frequency range is too broad. A steeper crossover filter at 35hz would be nice for integrating with large speakers. The bass is extremely clean and effortless at high SPL though.

50z crossover @ 30º:
db - hz - (deviation from 1khz reference; db)
69 - 20 - (-6)
72 - 25 - (-3)
74 - 31 - (-1)
75 - 40 - (0)
74 - 50 - (-1)
74 - 63 - (-1)
71 - 80 - (-4)
76 - 100 - (+1)
67 - 125 - (-8)
72 - 160 - (-3)
73 - 200 - (-2)

The info is appreciated. Am I correct in thinking that the -8 dB drop at 125 Hz is your main speakers? If so, that is a null at your reference location and can only be corrected by moving the main speakers or your seat position. Also, the -6 dB at 20 Hz, is that realistic for your sub to achieve? If not, it will not be terribly important for your overall listening since not much goes below that point at significant volume. Low frequency resonances are one thing, dinosaur footsteps are another! If you could adjust the drop at 125 Hz, then I believe you have an acceptable flat response. Any further adjustments may result in other peaks and valleys. I am interested to find out how things go. Good luck and have fun!

Briefly scanning over your post (I'm tired), it appears you have problems with getting your speakers/sub to couple between 80hz-125/160hz). You NEED TO make sure your speakers play solid at these critical bass frequencies, which are likely played through your mains from what I gather. Your Servo play well upto 80hz, or wherever you cross over. I recommend starting at 80hz, and work down if you like your music played lower through the mains.
The mains should be setup, from the listening possition, to couple flat down to the crossover ideally...same for the sub upto the crossover.
Without figuring your room dimmensions/setup, you should start by figuring where better places are for either your seating possition(s) or your speakers, so they couple at 80hz-160hz strongly! Looks like you got either the seats or the speakers sitting in some dips at 80hz,100hz,125hz, etc. You need to adress that.
If you have a single seating possition, and it's equi-distant from the sidewalls (center spot), I'd like to see you place the sub NOT DIRRECTLY LINED UP INFRONT OF YOU ON THE FRONT WALL! Either get it at 1/4 spots in the room conversely, or put on sidewall in middle. Never the less, if you place the sub were your final listening possition(s) is,where your ears will be located, you can play bass steady music and test tones through the sub, while you move around the room and listen! Where it sounds/measures the flattest/best, you can place the sub there! This works well.
In a smaller/medium sized room, I personally like to have a slighly lower response bellow 40/50hz, to compensate for the lack of bass absorption and excesss bass reverb in such rooms. It seems to make a better balance. In larger rooms, where there's enough bass absorption, (6000cu/ft or better), I prefer a more flat response down to 20-30hz range. That's my prefference for you.
So how big is your room?

The 125hz valley is purely the main speakers and listening position within the room. (the response from 63hz - 200hz is the same with or without the sub)

If I move toward the center of the room, the null shifts downward to 60hz, basically gutting the midbass and causing a large rise between 20 and 40hz.
20hz extension is no problem at all for the sub, though I often wonder how much impact it actually has on the quality of the bass response.

I can use a 120hz crossover, allowing the sub to fill in the 125hz valley, but this results in a shelved response, with the range from 20-60hz being 9db higher in level than the upper bass, and 7db higher than the reference level.
Tends to vibrate the light fixtures and windows audibly.
The 8'x5' living room window sounds like it's gonna collapse at high levels, and the character of bass instruments is not particularly realistic.

Flrnlamb,

Thanks for the detailed response!
The room is 13x17 with an 8' opening into a 10x10 dining room at the speaker end.
Here's a link to an edited illustration I made a few years ago for the tilesetter.
(white areas are walls; tiled area is floor)
http://www.byrographics.com/media/room-setup.jpg

Don't forget to add or subtract these amounts from the Radio Shack correction table to your figures:

Radio Shack SPL Meter Corrections
------------------------------------
10hz.....+20db
12.5hz...+16.5db
16hz.....+11.5db
20hz.....+7.5db
25hz.....+5db
31.5hz...+3db
40hz.....+2.5db
50hz.....+1.5db
63hz.....+1.5db
80hz.....+1.5db
100hz....+2db
125hz....+.5db
160hz....-.5db
200hz....-.5db
250hz....+.5db
315hz....-.5db
400hz....0db
500hz....-.5db
630hz....0db
800hz....0db
1k.......0db
1.25k....0db
1.6k.....-.5db
2k.......-1.5db
2.5k.....-1.5db
3.15k....-1.5db
4k.......-2db
5k.......-2db
6.3k.....-2db
8k.......-2db
10k......-1db
12.5k....+.5db
16k......0db
20k......+1db

THX Home Theater 16000 -8.5
20000 -11.2

Dave

Thanks.
The figures posted already have these corrections.

Let us know how you do with moving things around. I think that is the simplest solution. It is harder to tame high peaks, but your room configuration allows the escape of some of the high pressures, so I think your situation is quite manageable. If you like your speakers, crossover at the lowest point since the sub will not be localized and your front speakers will have a more coherent sound. Good luck!