Only you can decide whether or not any problems arising from contact to his father outweigh the benefits. However, in general I believe it is important for a child to have contact with both parents, to the extent possible and practical.
From your question it is unclear if there is behaviour by the father that poses a danger to your son (such as violence or neglect). You mention "drinking problems", which must be taken seriously, but can be controlled to some extent. Apart from that, the main problem is that you do not like the fact that the father does not accept his responsibility as a parent, and only does what suits him.
While this is a deplorable situation, I don't see how (further) limiting contact would make it better, either for your son or even for you - as long as the father does not show harmful behavior.
My personal advice would be:
- Try to encourage and support the contact the father is willing to have - not out of gratitude towards him (which you do not owe), but for your son.
- At the same time, do not feel obliged to go out of your way to accomodate the father. If he wants to visit at a time that is inconvenient for you, or do if he does things that bother you, set clear boundaries. Do not put up deliberate obstacles, but also do not feel obliged to make up for any lack of responsibility or planning by the father.
- If he later chooses to share more responsibilities, you can always re-asses the situation, and choose to accept this, or not, or conditionally.
- Keep an eye on the father's behavior and your son's interactions with him. If the father's addiction causes problems, you may, should and possibly must intervene. One can only hope the father gets his addiction under control, but that is not something you can influence.
That way, you maximise the good things your son can get from his father, while shielding you (both your familiy and you personally) from further problems.
Finally, do not begrudge the fact that the father is (sometimes) shirking his responsibilities. While this is a sad situation, there is nothing you can do about it. Just limit contact to whatever if beneficial for all of you, and let him make his own decisions otherwise.