Going anon here, because this is kind of personal, but it seems I was in your son's shoes some 20 years ago.
My father was visiting me only on my birthdays for the duration of the party. Since he started doing this, I was apparently developing mood swings. I'd go from hysterical laughing to sobbing to rage and back in seconds for apparently no reason. My mom was also told by my teacher that she would often find me crying in some corner of the school.
She took me to a psychologist and after some sessions, what she explained to my mom was that every time my father did this it was like meeting him for the first time and then having him die within hours on the same day each year. I'm not sure if he did, but if he happened to miss one, I'm sure the disappointment would have also taken a toll on me emotionally. I probably waited for him throughout the year. Anyway, my mom called him and told him to decide to either stay and live together or disappear from my life. He decided to disappear.
Fast forward to now, I can't say with certainty how that specifically has affected me. Lots of other stuff has happened. A lot of hardships since then, stuff that she tried her damnedest to shield me from, but couldn't always.
I grew to have a very, very close relationship with my mom, and I always understood her to have the roles of both mother and father. As I've grown and seen fathers, I've come to realize that she wasn't the same as the real deal, but I still loved her for doing her best.
People have asked me if I wouldn't want to meet my father again, and I've told them no. My father is a stranger to me. The only difference to an actual stranger might be that slight inclination to punch him in the face, but I know that he'd be too old to take it if he's still alive. My mom wouldn't have wanted me to, either.
I still hold on to the Nintendo 64 he gave me once, so maybe I do care slightly. It was the first (only?) thing he ever gave me.
Anyway, in conclusion: Be a father or don't be. There is no middleground.
EDIT: In response to the comments, I want to share my reasoning on the matter from a detached point of view.
A father (a parent) is basically a role model you can depend on. A child is an ignorant, defenseless creature that needs someone to rely on. The younger the child is, the more true this is. The child needs to know that he can count on his parent. When trouble hits, the child needs to know that out of all people in the world, at the very least he can count on his parents to always be there, exceptions being extraordinary or necessary for the child's well-being.
What is a parent that is only sometimes there? Well, I think that maybe he's a friend. An interesting friend that shares your genes. A relative?
Maybe you can establish that kind of relationship. I honestly don't know. I'm not a child anymore, so it's becoming increasingly difficult to think like one. However, it's been my observation that parent-child attachment is very real. Just recently, my child-cousin was completely convinced that her parents had abandoned her because she woke up in the car and they weren't there. They had been away for a minute while they carried groceries into the house. For her, there is no one more important in the whole world than her parents. She might take them for granted at times, but they never cease to be that important.
It's honestly hard to put myself in your shoes, because I'd just take responsibility. If conditions were so that I couldn't go all in, I suppose I'd just do however much I could. Support is still support (this isn't just money, btw), even if it's less than expected.
I think the important thing is to avoid failed expectations. Be clear to the child that he can't depend on you like other children do with their fathers. Be on the watch to see if your limited presence is affecting him negatively, and leave if it is. I really wish you'd stay instead to fulfill that attachment, but whatever. If this happens, I hope you could at least keep in touch with his mom to give whatever support you're able/willing to give in times of need. Being a single parent is very, very difficult, very tough.