I fully endorse the answer by Leopoldo Sparks. We don't have enough information to recommend a course of action, but from the few hints we have, being cautious seems to be well advised. Ideally the son can see the father, but it might be prudent to have some security measures in place, such as not letting the father be alone with his son.
But that's just repeating what Leopoldo already said. What I would like to add is that you (the mother) might want to consider therapy for your child. I know that your child is neither mentally disordered nor psychologically instable, but the fact that your son is – probably continually or repeatedly – "upset that he can't see his dad" might in fact cause him harm.
Some children overcome separation from a parent (by blood or adoption) easily, while others take this very much to heart and over time develop a psychological condition that can range from depression to aggression.
Though I don't work with children, I am a psychologist and have many colleagues who do. In our psychological outpatient clinic, there are quite a few children who have no other "problem" than that their parents are divorced. Sometimes they suffer from the quarrels between the parents, sometimes from visiting arrangements that disrupt their own private life (such as seeing their friends). None of these children are psychologically "sick", but all of them suffer from life circumstances that overtax them. And they all profit from an outside ally, an adult explanation, and strategies to deal with their own emotions.
So if you have the opportunity to let your son see a qualified child psychologist, I would recommend that to you.
All the best!