I 've lived in Tokyo for about three years and the best shops (vinyls) I've found were in Ikebukuro (one is located in the east side of the station and the other is in the west) ; records are not cheap (jazz for about 2000 yens; rock for about 1500 and more for collectors) but they were like new.
You should give a try to Recommint in Nakano (north exit, shopping mall, second floor); they sell almost only second hand CDs and you can even find some pretty good titles for 300 yens (or1000 yens for 5 pieces during the week-end) if you're lucky.
There are also big sales in September and October organized by Wave (kind of HMV) that gather records and Cd shops from all over the country. It usually lasts one or two weeks and you can buy good stuff too.
Disc Union in Shinjuku and Kinchijouji are popular places too.
Hope this helps,