I enjoy that experience also. In fact, most of my vinyl collection is from second hand shops.
It's hard in the bay area (CA) to find the $1 albums because there are audiophiles that usually beat me to it, I think.
But at Ameoba music, and Streetlight I can usually find good recordings that are worth listening too. Sometimes, I'll find a true gem - (good performance, great sounding recording).
at places like Goodwill, I usually get about 1 or 2 out of 8 that are worth listening to. I toss the rest. That's the thing about vinyl; its either good or bad - not much in between.
Fortunately, I like classical music. There is still a lot of that available. Jazz I think would come in second. I don't even try to look for Rock.