TL;DR: You can fix this, and do even better than that. You can become a great dad!
We always want our children to learn the consequences of their behaviors, believing that it's one of life's most valuable lessons. Now it's your turn to face consequences. There are no quick fixes.
You have given your eldest ample evidence that he should be afraid of and wary around you. To be desperate to change that is also to be unrealistic, and it basically means you want him to throw away the valid feelings he has for the risky proposition of trusting you again.
All parents make mistakes. I've made my share (maybe more). When I ask my (now adult) children what they remember of some of my worst behaviors, I am so grateful to hear that usually they don't remember anything. They do remember some of my apologies, though, and they laughingly remember some of the crazy restitutions I made when I was wrong (restitution is a big issue for me).
So, all that to say what you've done, whatever it is, may be fresh in his mind now, but it won't stay that way if you don't reinforce those memories anymore.
While I agree that parents aren't responsible for all of a child's personality, you have modeled anger as a coping device for him. You admit that you've smacked him and shouted at him, and he's (justly) wary/afraid. Sure, you didn't say "This is how I want you to act: get angry!" But kids do learn coping mechanisms from adults.
If you haven't done it already, I would advise you first to sit down with him and apologize (profusely) for what you've done to him that was unjust (knowing that not everything was so). Don't worry about putting it into child's sentences. He will likely understand even a fairly adult apology. Own what you've done, explain why it was wrong, explain how you now know why it was wrong, and ask him if he can forgive you (please don't demand it). Promise him you'll be more patient and wise from now on.
Next, stop yelling. Yelling is threatening to people who don't feel powerful, and he's had enough of it. Express your expectations in a calm and rational way. Use a method of discipline that allows you to emotionally disengage from the process, so that you won't get angry. 
Work on letting him express his feelings to you about things in a safe way. You can start with less threatening, positive feelings. Pay attention to what he says. Act on it when you can. Work around to negative things. Teach him feeling words so he can be accurate. Words like frustrated, grumpy, uncomfortable, worried, confused, safe, excited, proud, thankful, disappointed, lonely etc.  This will help him to identify his feelings and will present an opportunity to model how to deal with these feelings in a safe, wise way. Allow him to articulate negative feelings about your behavior; he has them whether they are articulated or not. You don't need to do what he wants you to do about them (he is only 6 after all), but you need to show him he's safe with you.
Finally, forgive yourself. Apologize, own, ask for forgiveness, and grant yourself some as well. We all make mistakes.
If you do these things, you will do far more for your son's well-being, his future, and his relationship with you than you can possibly imagine right now. Time does heal many wounds. Love in practice heals many, many more.
Below are more resources. Keep working on that relationship. You'll all win in the end.
 My go-to book for effective discipline is 1-2-3 Magic (appropriate starting at age two), one of most effective approaches to behavioral (self-) correction I've ever encountered. When applied correctly and consistently, it allows time outs to be applied without the parent losing their cool, while it gives your child (if not right now, it will early on) an opportunity to correct himself if he is able to before the time out, gaining some experience in self control and managing frustration without loss of self esteem. (1-2-3-Magic: Effective Discipline for Children 2-12 Thomas W Phelan PhD.)
 You can use appropriate words from lists here.
 Helping Young Children Manage the Strong Emotion of Anger
 Teaching Young Children Self-Control Skills