so you've set the b&k to "no center" and, when your sources are connected to the analogue ins (whether rca or 5.1)the b&k is acting correctly. however, when you're sources are connected digitally, for whatever reason your b&k is reading "no center" incorrectly and is sending the center info to the center channel (except digital connection in stereo mode, when the b&k acts correctly). ergo, it seems that your b&k processor is malfunctioning when it's in "surround mode"--this is evidenced by the fact that even tho you've set it to "no center" when you connected a center speaker the b&k actually sent the center info to the center speaker.
if you'd humor me, please:
try using your current setup w/digital connection in "thx" mode--do you still get no center info?
if our conclusion is right, two possible workarounds:
1. get a center channel, since the b&k seems to be sending center info there even though you've setup for no center; or
2. use the analog inputs instead of the digital ones since these seem to work correctly; you could add a dac to your sources to improve the sound.
interesting problem--please report back