US clocks fall back on Sunday: why does one hour mess with us for days?

Daylight saving time ends in the US this Sunday, Nov 1, a week after Europe. The "extra hour" sounds harmless, but the research on time changes keeps pointing at our circadian clock being slow to re-sync to a shifted light/dark schedule, and sleep scientists' groups have argued for permanent standard time rather than permanent DST because morning light matters more for anchoring the body clock. The spring shift gets most of the attention for heart attacks and car crashes, though. Does anyone actually notice the fall change in themselves, or is it only the spring one that hurts?