Bits Are Bits, Right?

So I'm currently heading down the path of exploring which CD-Rs sound best in my CD player, along with what burn speeds sound best and what CD burners make the best CDs. I already know from my research that the more accurately the pits are placed on the CD (e.g. less jitter in the recorded data), the better chance I stand at getting the CD to sound good. There is a counter-argument to this idea that goes something like this: "Bits are bits and as long as the CD player can read them, the accuracy of the spacing doesn't matter because everything is thrown into a buffer which removes the effect of any jitter written into the data during burning." I know I don't agree with that logic, but for the life of me I can't remember the technical reasons. I know I used to know. Haha! 

So who here knows why buffers don't solve all of our problems in the digital realm? How come timing accuracy matters in the stages before the data buffer?