I agree we could use more background, and will edit this answer as more is posted.
You have a perfect right to be upset and worried; upset because you did all the parenting without an iota of help from the father, and worried because the father doesn't have a good track record of being a loving, supportive and responsible person.
However, this is about what your son a) needs, and b) wants. You know your son better than anyone else. If he has expressed thoughts that indicate he felt abandoned/unloved by his father's lack of involvement in his life, this is an opportunity to mitigate this feeling to some extent (probably not greatly - abandonment issues run quite deep for most - but maybe so; his father might be a much better man now.) Your son deserves the opportunity. I will not comment on what his father deserves, because this is about your son, not the father.
Your son may need this desperately, not at all, or anything in between, but at 11, unless there is abuse involved, it should be his choice. It's what he wants to do, or he would have expressed a reluctance to do so.
I don't know what country you're in, but the father may have a right, even at this late date, to contact his son as well.
My advice is that you exercise caution quietly, and withhold any strongly negative opinions you may have about his father, about his father's treatment of you, or about the fact that he wanted his son to be aborted. That kind of information doesn't help your son to deal with his loss at all; it only makes you feel heard (which is understandable; you've been through a lot.) However, hearing your feelings is not your son's job; express negative feelings and concerns to a friend or family member, but please protect your son from feeling even worse about his predicament.
By quiet caution, I mean pay careful attention to anything your son reveals about the meeting(s), his mood and behavior after visits, changes in his school or social performance, anything that might indicate that the visits do more harm than good. Give him an opportunity to discuss his feelings, whatever they are, because he needs the most important person in his life to be able to work this out with. If he has positive feelings, he may be reluctant to share them if he knows you have strongly negative feelings about the man, so as difficult as it may seem, try to express happiness for him if he is tentatively positive in his discussions. Encourage without bias all honest discussion, positive, neutral or negative.
If these meetings seem to meet some need in your son and don't seem to cause any harm, let them continue. It will be showing a great deal of respect for your son that he will appreciate not only now but when he has children of his own.
If the visits seem to be causing any harm, get involved. Again, talk to your son about his feelings. If it's just that he's conflicted, he might do well to have a therapist to talk to also. Abandonment issues are not easily resolved.
If he seems to be actively harmed by or expresses that he doesn't want to continue the visits, stop them and talk to a lawyer.
As to how to 'handle' the father, that depends on how the father treats you and your son. You owe him nothing, but the kind thing to do for your son's sake is to treat him politely. If he's unkind to you but kind to your son, you can have extremely limited contact (just a brief meeting at the point of exchange). If he's kind to you, how much further you want to go (e.g. letting him come into the house when picking up your son, etc.) is completely up to you. If you want to entertain your son's thoughts on this, remaining open to suggestion is great. However, this is really your decision, not your son's or anyone else's.