Of the big four mentioned above, Neil Young, Leonard Cohen and Dylan and Jerry, Neil and Jerry are the ones I can listen to throughout entire career. When Cohen and Dylan began to sound more like croaking frogs than human is day I couldn't take it anymore.
There is this one obscure female hillbilly singer from 1950's, can't recall her name for life of me, somewhat recent thread here mentioned her. Anyway, the voice is probably the single most unbearable voice I have ever heard!