I rarely use center channel with processor

Does anyone else not use a center speaker because it is better to use only the mains? 

Definitely!  One should use the center speaker only when the source material has discrete center channel content.  OTOH, don't eliminate it if there is such content.  Such manipulations can achieve only chance, sporadic success.

If you are talking about playing music through your home theater system, then just allowing it to act as a stereo can sound better. If movies, then probably not. In movies more than 80 of the dialogue comes through the center channel. and you're looking to recreate something usually created in 360 degree sound. 

PS - I've experimented with the use of analog matrix systems to extract a center, including Dolby Surround/ProLogic and PL II.  Among them by far the best sounding for music was the DTS Neo:6 system.  Really sounded great and with a center channel gave very realistic imaging. 

Of course, use what you'd like, the point is enjoyment. 

I think maybe what it gets down to is the nature of how the recording was done. If the engineers did a good job with five channels and isolating voices in the center channel then it may work out. But if they mix a lot of non-voice stuff in the center channel and let a lot of voice stuff inappropriately mix into the main channels it mixes things together in a bad way for sound purposes.

Also if you have a system that resolves detail really well that's really impactful too and you notice these things.  

So it gets down to quality of the recording and whether you truly have a good multi channel recording.

  Back in the day my surround had JBL fronts , Klipsch Heresy II rears and Klipsch RC center and a pair of subs. I had a very early Oppo that played SACD. If I used a standard cd , I set it to stereo. I collected concert DVD’s and stereo through 2 or 4 speakers was preferred. In retrospect I ONLY used the center speaker for movies.