Why are subwoofers so polarizing?

Because they are often problematic to get set up well and essentially an optional accessory and not necessarily always a core part of a good sounding system.

There are three scenarios one can follow regarding subwoofers:

1) pass on the lowest octave of sound and get the rest working really well, which is typically the easiest approach to getting good sound.

2) buy a more "full range" design that can cover the lowest octave alone (essentially subwoofer is already integrated by the designer)

3) buy a separate sub and set it up/integrate it yourself.

Options 2) and 3) both of which address the lowest octave by design can be a challenge to get set up really well in any particular room.

So really its all that is involved to get the lowest octave of sound working right along with all the rest that is a challenge in general. SUbs just provide a separate device, but the problem is still often the same and may or may not be any easier to solve case by case.

There is no simple one size fits all solution. Does that make it "polarizing" maybe?

When I use a sub, I try to set it so it sounds right with the more bass heavy recordings first. That means it may do little or nothing with other recordings that do not have much going on in the lowest octave. IT seems to be a reasonable and relatively easy strategy to take that works well for me. The goal is to help when appropriate but not try to make something out of nothing if its not there in the first place. Good full range recordings can benefit greatly from a well set up sub, but many recordings may benefit just marginally or not much at all really.