I applied the same basic concept during the packing process the last two times I moved. As I handled each item, if I couldn't think of a really good reason within 5 seconds to pack it in a box, it went in the give away/throw away pile. By the last move, all my musical media was digital, so at least that choice was already made. It's a lot easier to keep ones and zeros that you never listen to!
Throw it out!
I just discovered a technique that will save a lot of aggravation. When I find a record that is not satisfying me sound-wise (or sometimes musically,) I dispose of it or relegate it to the inactive file. Rather than having it available in my collection to aggravate me, it gives me great satisfaction to get rid of it!