There is very little I can say that hasn't already been said.
Disclosure: I don't have a son (yet, but that's another story for another day) but I have kids.
My point being that if I did have a son there will inevitably come a day where he would want to play with his sisters' toys. There is a clear distinction between behavior I would define as okay for a brother with his sisters and behavior that needs correction. Me personally, I wouldn't let that behavior continue too far beyond 5 or 6 years old, and I'm pretty sure I wouldn't let him play with them by himself, but that's just me and my wife. I would also say that if there were no little girls in the house, there would be no Barbies/Girls toys for him to play with either. You mentioned he was your only child but didn't mention where the Barbies came from, or where he's at when he plays with them, so I'm curious.
No one on this site knows your son better than you, and you must observe your son and see if he's showing behavior that needs to be corrected. Anyone can tell you not to worry, but we're not there when you see him interacting with the Barbie. He could just like girls (and a Barbie is like a girl, to a 5 year old anyway), or it could be a sign of something deeper. Part of the job of a parent is to correct our children's behavior if need be. It sounds like you do a great job already of observing him and how he acts, you should keep it up. If he displays behavior that makes you uncomfortable, then you can change it. You know how you are and how his father is, so you should be able to spot behavior that can turn into a problem if it continued on it's own. The good thing is at his age now his attention span is so short that you can easily expose him to other things to keep his mind off of the Barbie dolls, for example.
Finally, you didn't ask about this, but children should have both parents in their lives if at all possible. I'm not saying send your son to live with his father permanently, but your son at least deserves to continue learning who his father is, and at this age if the father wishes to spend time with him, then (if there is no legal standing preventing it) you should continue to let the son visit, or vice versa. Your son is old enough to tell you what happened at dad's house and if it makes you uncomfortable then you should handle that. Eventually, if there comes a day where your son no longer wishes to see his father, then at that point it's up to your son.
Update in response to comment
I don't mind the d/v, my beliefs usually attract those on this site, but since I was asked, to answer your question, I didn't say the boy shouldn't be allowed to play with dolls. I did say that I would monitor and correct that behavior if I felt that letting it continue would not be in my sons best interest.
To elaborate, in an answer above there is a paragraph that starts
If you let his father take the reins on this.... I respectfully disagree with that paragraph completely. I will agree that his father might not be the best person to handle the situation, but teaching a child that something is wrong is not universally detrimental to that child's mental well being. Children are moldable by their very nature, and it is the parent's job to mold them. Simply put, if a child knew what they wanted, and knew right from wrong, then they'd be wise enough to make their own decisions, but they don't, so they're not.
To be blatantly specific, if I felt my son was starting to become confused and not understand there is a difference between boys and girls (not the difference, just that there is a difference), then any behavior that contributes to that confusion needs to be monitored and corrected if necessary.
Let's be honest: The OP is worried her son may grow up effiminate, or worse, homosexual. The father is as well. It's obvious from the question, and it's an understandable fear. How people answer this question will be largely (if not wholly) governed by their views on homosexuality in general, which is okay. Actually, I largely agree with the accepted answer of Wirehead's, but wanted to expound because I feel that it's less black and white and more of a gray area.
I obviously disagree with statements such as
..Heterosexual and homosexual attraction are simply the result.., but I didn't downvote, because everyone is entitled to their own beliefs, and it could have possibly turned into a flame war. Back to the topic though, I do not believe that boys playing with barbies will inherently make them effiminate or homosexual, although I do think encouraging female behavior in a male in general can lead to this if left to continue.
This belief is more from personal experience than it is from being a Christian. I know someone firsthand who had two sons, let the oldest dress up as a girl, encouraged this behavior, and, once the child was a teenager(not of legal age), gave him estrogen pills. After seeing this, his little brother has now followed in his footsteps, including the pills. The father is around, and I have no idea why he didn't object to this. In my opinion this should be illegal, but I digress.
I believe this particular example is the result of behavior that was damaging to the child's mental health being encouraged. That is an extreme case yes, but I believe it's relevant to the OP's question because of the ultimate result, and also furthers my point about detrimental behavior being encouraged or allowed to continue.
Once again, Let me further be clear: I don't believe a boy playing with his girl cousins' barbies with his cousins is harmful, to a certain extent. This is no more harmful than jumping rope or playing hopscotch. However, I wouldn't let those girls paint his nails or put make up on him, which could cause confusion.
In addition, there are plenty of 'gender-neutral' toys and games that every kid should have, 'gender-neutral' in this case meaning toys that don't carry the gender stereotype with them (barbie dolls, for example).
Moreover, of course, the question may come up about letting girls play with toys that are traditionally played with by boys. I don't have a problem with this either, but again to a certain extent. I'm a little more relaxed about this for the simple reason I've seen more girls that were tomboys when they were little turn out normal than I have effeminate boys, although it does happen. I know this will warrant downvotes from those that agree with the homosexual lifestyle in general, but I'm not in this for votes: The OP asked a subjective question, so I gave an answer based on my beliefs, as did everyone else. No one here can say without a shadow of a doubt that what will happen one way or the other, we're all just giving insight based on our own life experiences and beliefs.