I was in a band with a guitarist (Paul Skelton. He plays on albums by Austinites Wayne Hancock, Libby Bosworth, and Cornell Hurd) who loved Tom's early, "beatnik"-era albums, but I found them a little too self-consciously hipster for me. His Swordfishtrombones album changed all that, and his next (Rain Dogs) sealed the deal. Great albums!
But then he took to singing through a bullhorn, and lost me. In the 90's I saw him live in a double-bill with Los Lobos (how's THAT for a show?!), and found him not to my liking. I still like his acting, though. Down By Law, and Coffee And Cigarettes (in a scene with Iggy Pop!) in particular.