It is exhausting when you first must get used to having a child around all the time. For someone who is used to having a substantial amount of "down-time", it can be devastating.
There are some people who get uplifted from being with all children, and some who get uplifted from being only with their own children, and others for whom being with any children is an emotional drain. This is due to a difference in personality. Just as some people find being around others emotionally draining, and others require it for their emotional health.
For me, it was an extremely difficult transition to make. I was used to having several hours a day to unwind after work. I would read, write or play computer games. I do not go out and socialize; being around people is exhausting to me, even if they are people who I like very much.
After I became a mother that was no longer available to me. At times I was so depressed I could barely get out of bed. Here was this noisy, needy little person who never went away. Even when she was sleeping there was always the knowledge that she could wake up at any time, so you could never relax. All my needs for alone-time had to be put in a locked closet. It was like going weeks without sleeping, only the exhaustion was emotional rather than physical. And even though there were moments when I was not needed to attend to my child, it was like closing your eyes for only ten minutes at a time. I could not simply relax, especially when I didn't know how long I had before my child needed me. I was always "on call".
The only way you and your wife will be able to have the downtime that you need is if you help each other. Schedule a "daddy day off" or "mommy day off" (be sure it's a fair exchange) where the parent who does not have a day off agrees to take the child and leave the house for a set schedule.
The reason it is important to leave the house is that when your child is in the house you feel responsibility and guilt for not responding to their needs. Removing their physical presence reduces that. And the child will not feel that he is ignored if he is not with you. The person who is having a "day off" could also leave, but for me it is more relaxing if I can be in my own home.
Once a year my husband takes the kids to visit his parents for a week in another state, leaving me with an entire week alone in the house. It is very refreshing to my spirit. We also occasionally would leave the kids at a daytime daycare place for a few hours on the weekends, It was expensive but a good way to get things done, for things that are difficult to do with a child.
It seems to me that what you are looking for is a reason why it is good for you to have some time to yourself. Do not feel guilty about needing time alone. Everyone is different. Some people need more of that than others. Just because you have become a parent does not mean that all of your needs must be sacrificed.
To me, it is like when you are on an airplane and the flight attendant gives her or his speech. At some point they tell you that in case of emergency, oxygen masks will drop. You must first put the mask on your own face before assisting your child.
This is not selfishness. It is practicality. You cannot assist your child if you pass out for lack of oxygen. For me, it is also the same with food. When I get home in the evening the first thing I do is eat something, before I prepare the family meal. I have hypoglycemia, and if I allow my blood sugar to get too low, I become incapable of caring for my family.
If you are unable to have some time to yourself, you risk building up resentment toward being a father. Even if you try hard not to, even if you know you should not, it is difficult to always make our emotions obey us.
Remember to give your wife what you ask of her. Even if she says she does not need it, it might be because she feels too guilty at leaving her child. Tell her that you and your son are going to have daddy-son time, and are going to do something fun. Enjoy yourself and have fun with your son, and she will be able to relax too.
I have found two things are important for a good "time out". One is that exact times are kept. If you say that you will be gone from 10 AM to 4 PM, don't come home early unless there is an emergency. The second is that there be no requirements for the person taking time off, and no direct communication from the person on. Texting is okay, calling is not, except for emergencies. And do not ask "while you are taking time off, can you --do some task--?"
One of the nicest things about a time-out is at the end of it, when you have been away from him for many hours, you are maybe feeling like you have missed him and you can enjoy his company more, while your spouse relaxes.
I am sure that you will be able to work something out to fit your needs and your wife's. You just need to talk about it, and be honest about what you need, and willing to give her what she needs as well. It sounds like you are both so focused on giving your son all that he needs that you forget the needs of each other.
One last comment. It may be that your wife has needs other than time off that she wants. Something she wants to buy, that you would be disapproving of. Or maybe she wants a night out with you, where you go to a movie and restaurant and hire a babysitter for the night. It is okay to have different needs, and it is okay to bargain for what you need. Give each other that permission.