Usually the sweet spot is how the speakers is without a port blocked.
Ports are designed specifically by length and diameter to extend the bass response by a "tuned" tube that resonates at a specific group of frequencies. You are a speaker engineer and your speaker plays naturally to 100Hz. Put a port in that has a center resonance (or bass boost) at around 85Hz, the total speaker now sounds like it goes lower than the 100Hz, the port is "extending" bass response downward.
Think of a long tube you blow into at a football game and make a loud low frequency howl- this is the port tube demonstration. The frequency of the boost depends on the length and diameter of the tube. [Passive radiators do the same thing]. Allow this tube to boost everything moving through the speaker and it sounds like it has lots of bass when the main driver isn't producing that much bass at all. This is the bose trick that works brilliantly in some of their models.
Stuff the port and you usually move the roll off (the lowest frequency the speaker reproduces) UP, reducing total bass from your speaker. In one speaker I import, it comes with foam port plugs and was designed to use this port stuffing feature to make it more linear for those that prefer this sound vs those that prefer a little bass bump down low to sound better on pop music or EDM or whatever. When the speaker is designed with this in mind, its a good thing. But this is rare- I have heard of few speakers where port plugs are part of the feature set.
Many cheap speakers are so poorly designed that the boost from the port is huge, excessive, because manufacturers know people buy the speaker "with more bass". So in this case, plugging the port makes the speakers sound totally different- and may remove all the [perceived] bass and suddenly make your speaker sound thin and all midrange and top end. These areas are generally unaffected by ports and remain the same no matter what the port does or does not do.