And why is that more of a distraction than yakking to a passenger ? 

I see some drivers looking at their passengers instead of the road while talking, which is stupid but some people do it.  There is no need to look at a handset.