It is likely that the walls are acting as drum skins. They seem to have similar resonances, from your accounts.
This is not going to be an inexpensive fix, unless you engage the right expert.
Your problem will be in finding a functional fix as few folks know what they are doing in the case of ultra low frequencies. This is the hard reality of sound and noise control. It being so prevalent that the measurement weighting standards totally ignore these sounds, as they don’t know how to deal with them.
Or put the fence back up.