Certainly feel OP's pain here and believe he is being generous in his responses to buyer. Had similar on a deal on another forum which left a sour taste.
My experience with Paypal is that they side with buyer everytime there's a report of item 'damage' to save themselves hassle of dealing with buyer's credit card company, but an important part of this is to report supposed shipping damage to Fed Ex. They will usually send someone to view the boxes at recipient's location to determine if the damage was possibly shipping related, and if so they can help resolve it, but even if not, this is important data for further interactions with the buyer and can make some who claim shipping damage soften their stance.
Now it is still possible the buyer is being straight up and the damage did occur after you sent it but before he received it, but I'd get the shipping company's examination on this just for the record. As shipper you can initiate this and they'll contact the buyer to arrange inspection. The suggestion to send pics of corner damage to Harbeth is a good one too.
From what you describe it doesn't seem possible that the damage described was done in transit. It sounds like someone tried to pry the grills off as you said. It really comes down to whether you want to take an unjustified hit to your rating or pay for something that appears to have not been your fault. I would contact Harbeth and hope they show some sympathy and help you out. They might, especially if your replacement speakers were another pair of Harbeths.
I’ve owned Harbeths and removed the grills once gingerly prying them loose. Harbeth wants you to use the grills judging by the difficulty of removal. The inner and outer boxes provided by Harberh for shipping are very good. The outer box might sustain damage, doubtful the inner one would. How the inner boxes would suddenly “open” and cause damage is inexplicable. I call BS on the buyer, so I would ask him to return the speakers to you with him paying freight and insurance or have him deal with FedEx. The standard damage with FedEx is $100 so alternatively you could reimburse him the difference for the grill prices. My bet is that the grills are expensive.