Since you're asking in regard to a toddler, my answer is this:
It's not a toy. Make no exceptions.
I would very likely allow older kids to play with my iPhone, because they're more likely to take good care to not damage it, and to not make calls to random people in the phone list. But a toddler doesn't understand what he's doing, he's just touching semi-randomly and while enjoying the changing colors.
The toddler of a friend of mine ruined an iPhone because he, too, was incredibly attracted to it and eventually physically broke it. (Turns out an iPhone doesn't do very well as a car on a rough stone floor. Apparently it lacks something called "wheels." Who knew?) My wife and I are very protective of our computers and iPhones; from the very start we made it clear that these are our devices and only we may use them. Despite being very active and inquisitive, out toddler learned to respect that electronic devices are not kids' toys but adult tools.
My recommendation to you is that you begin teaching your son that a phone is delicate and expensive, and not a toy. Until he learns to not play with it at all, make no exceptions. Do not under any circumstances let him touch or hold it. It's forbidden, just like a kitchen knife. In your case this is going to be difficult because he has already learned that he is allowed to use it. Persevere.
Make it a new rule that he is not allowed to touch your phone. Be consistent. As long as your don't break this rule, he can learn that it really is a rule. If you break your rule, you're teaching him that he can break the rule too. So don't give in, no matter what. Really, treat the phone like a knife.
Only when he demonstrates that he has learned to respect your things can you allow him to play with it under supervision.1)
Please note that my context for this answer is that an iPhone/iPad/laptop is an expensive piece of equipment that I can't afford to replace on a whim. I'm sure suitably affluent parents have other/higher limits, but my limit is lower. I'm less strict concerning less valuable non-toys.
1) Software recommendations generally aren't in the scope of this site but it fits into this topic: There's a useful iPhone app called "IncarcerApp" that will disable the home button so that the user remains in the chosen program. My toddler is sometimes allowed to use a painting app, locked down using IncarcerApp, and only under supervision.
(Edited to emphasize my context relative to other answers.)