High power amp, does it matter at low volumes?

I just woke up to a wealth of comments!  Thank you all, now, to digest a few new concepts.

I also realize that I made a mistake, my amp is Mark Levinson 532H, not a 522H.

I will be drinking my morning coffee and rereading and digesting.  A lot of interesting things to consider.

The amp I used before this was a Bryston 3B NRB, and there was a large jump in performance when I installed the Mark Levinson 532H. ( which I gather does not double down ).

I echo @welcher analysis. It is instructive to do the calculations given your listening habit and environment to get some ball-park idea. That will tell you whether quality or quantity of watts is your limiting factor. I went from a Rotel 1572 integrated amp with 2x120 W to lower wattage Pass XA25 and sound improved.

The other question re dynamic range is frequency range your amp needs to handle. The biggest power consumption is in bass, and you can outsource that bass power requirement to dedicated sub(s).

The other question is re channel separation. One may make the argument that dual mono (or bridged) lower wattage amps is better than a bigger stereo amp. I run a Pass XA25 mentioned by @jonwolfpell and waiting for a decently priced offer to add a second one. [$5K second hand as advertised repeatedly on AG is a non-starter]. 

I used to own N803. Progression with the amps was as follows - McCornack DNA125, McCormack DNA225, Pass X250.5. 
Each step up in the amplifier quality increased the performance at lower levels. I lived in an apartment back then and we had a newborn. And neighbors. Late at night at low volume you get excellent sonics from B&Ws but obviously you don’t get the woofers moving air the same way they do when you crank it up. Go for the best possible amp you can afford that produces high quality high current power.

Look into Coda amps. Killer performance for the money. Pass X260.8 mono blocks, or X350.8 stereo with a good preamp would be a nice match. You need lots of effortlessly delivered current for the B&Ws. 

One other important bit with the Pass Labs X260.8 mono amps you get 34 watts of Class A power into 8ohms. This means, if I am not mistaken, 17 watts of Class A into 4ohm speakers. This is enough high current to never exit Class A with your B&Ws at lower volumes. Depending on a room size and what you define as low level listening but it will most likely stay in Class A the whole time and will start moving the needle into A/B at around 80-85db if not higher. 

Per Lanx0003 my amp seems to have a rated THD+N of .05 or -66 dB. A stereophile review showed it was actually .0019 or -94 dB Cold and .006 or -84 dB Hot.  Crosstalk is over 100 dB below 1 kHz and between 81 and 85 dB R-L and L-R.   Not the greatest specs..

I downloaded a sound meter app, I understand it runs on an Android, so accuracy is really an indication only. At normal listening levels I didn't exceed 65 dB at my listening position. Okay, some times I crank it up, but rarely. 

My speakers are rated to be 90 dB efficient, and I sit 10 ft away from them.  You are supposed to need a minimum of 50 watts upto 500 watts.