eBay counts attempted delivery as delivery, so you've done your job. As a courtesy, after the item is returned to you, you may offer him a refund of the Item Cost (less postage cost).
If it were me, I would add him to my Blocked List and stop messaging outside of an actual Case. When people are angry about making a mistake and looking for someone to blame, protracted messaging usually harms more than helps.