Impedance in headphones vs. speakers - confused

I would really appreciate it if someone could explain the difference between impedance in headphones compared to speakers? I'm confused from the research I've done since it looks like the higher the impedance in speakers (such as 8 ohms) the easier they are to drive. Whereas it looks like the opposite in headphones. A 32 ohm pair of headphones (such as most headphones used for iPods) is easier to drive than say a 600 ohm pair which usually requires a headphone amp. Is that right?