High power amp, does it matter at low volumes?

I am a low volume listener as well, but I have not experimented with low wattage amps. I was not able to find any info on your amp, so I can’t gauge how other amps might improve on it. My feeling is that you can only go so low, even if on paper you are not pushing the amp, and the lower you go the more important that doubling of wattage becomes. As others have stated, the quality of the watts matter, and it is pretty easy to best any HT integrated with decent two channel separates. 

I just woke up to a wealth of comments!  Thank you all, now, to digest a few new concepts.

I also realize that I made a mistake, my amp is Mark Levinson 532H, not a 522H.

I will be drinking my morning coffee and rereading and digesting.  A lot of interesting things to consider.

The amp I used before this was a Bryston 3B NRB, and there was a large jump in performance when I installed the Mark Levinson 532H. ( which I gather does not double down ).

I echo @welcher analysis. It is instructive to do the calculations given your listening habit and environment to get some ball-park idea. That will tell you whether quality or quantity of watts is your limiting factor. I went from a Rotel 1572 integrated amp with 2x120 W to lower wattage Pass XA25 and sound improved.

The other question re dynamic range is frequency range your amp needs to handle. The biggest power consumption is in bass, and you can outsource that bass power requirement to dedicated sub(s).

The other question is re channel separation. One may make the argument that dual mono (or bridged) lower wattage amps is better than a bigger stereo amp. I run a Pass XA25 mentioned by @jonwolfpell and waiting for a decently priced offer to add a second one. [$5K second hand as advertised repeatedly on AG is a non-starter]. 

I used to own N803. Progression with the amps was as follows - McCornack DNA125, McCormack DNA225, Pass X250.5. 
Each step up in the amplifier quality increased the performance at lower levels. I lived in an apartment back then and we had a newborn. And neighbors. Late at night at low volume you get excellent sonics from B&Ws but obviously you don’t get the woofers moving air the same way they do when you crank it up. Go for the best possible amp you can afford that produces high quality high current power.

Look into Coda amps. Killer performance for the money. Pass X260.8 mono blocks, or X350.8 stereo with a good preamp would be a nice match. You need lots of effortlessly delivered current for the B&Ws. 

One other important bit with the Pass Labs X260.8 mono amps you get 34 watts of Class A power into 8ohms. This means, if I am not mistaken, 17 watts of Class A into 4ohm speakers. This is enough high current to never exit Class A with your B&Ws at lower volumes. Depending on a room size and what you define as low level listening but it will most likely stay in Class A the whole time and will start moving the needle into A/B at around 80-85db if not higher.