This is mostly a legal issue, perhaps quite unfortunately. Because this man is your child's father, he likely has legal rights to see his child, and may push for those rights to be enforced if denied the chance to see his child.
You can challenge it in court, but I don't know how far you get - we are not legal council, and cannot offer you legal advice.
What we can offer you is some ways to cope with this, whether or not there is legal action involved.
First of all, you need to consider how much of a danger this man is to your child - and if he is a source of physical and voilent danger to him, then you should start seeking your own legal council for the safety of your own child.
If he is not a physical threat to the child, but you worry about the influence he may have, the best way to counter that is to be a positive influence on the child yourself, and to surround him with other positive role models. You, your friends, and your family can all have a great impact on the child, moreso than a single negative influence can overcome.
If you are worried about what he may do during his legal visitations of the child, and that you may not have the ability alone to prevent it, consider getting somebody to be there with you during those times - a friend or a family member who you can trust. This is especially important if you feel that the man is a danger to your child, but cannot legally prevent him from having visitation rights.
In short, your primary focus should not be on whether or not to bar this man from seeing your child - that is a legal battle, and we cannot advise you on how that will turn out or what you should pursue legally in that regard. What you should focus on is giving them a positive role model to follow, and in protecting your child in any situation that may arise.