All the research I've seen shows that it is the conversation itself that is distracting and not the physical act of holding the phone - the equivalent of driving whilst over the drink drive limit!

This is an interesting article.

http://well.blogs.nytimes.com/2008/12/01/chatty-driving-phones-vs-passengers/?_r=0

"Nearly every driver riding with a passenger completed the task – sometimes the passenger alerted the driver to the approaching rest stop. But half of the cell phone chatterers missed the exit, according to the study, published in The Journal of Experimental Psychology: Applied. “They drove right past the exit without even stopping or noticing,’’ Dr. Strayer said."