If I'm listening to the main system or headphones it's usually critical, or at least engaged listening. Unless I fall asleep that is.
There's plenty of other places for casual listening. Family room, bedroom, another bedroom, shop, or garage. Throw in the vehicles, too.