First, emotions are really big beasts inside us, and they can be scary things for any small child who doesn't feel he has the means to control them. Please keep in mind that your son isn't born knowing what to do when he feels angry or frustrated; these are just huge feelings inside his body and they feel really, really bad.
Your child will learn to cope with these feelings by modeling your behavior. If you yell and hit when you are angry, your child will yell and hit when he is angry. Of course, the irony is that you are angry and frustrated to such a degree precisely because he is not behaving properly when he is angry and frustrated.
Of course it is easy to be calm and objective when your child is not having a meltdown, but in the heat of the moment, it is not always so easy to control our own behavior and really think about what we are teaching our children. It can be especially difficult if his screaming fits wake his brother (I have two boys as well, so I understand). So the very first thing you need to do is deal with his tantrums constructively, before they escalate to point where you lose control.
He is five, so pretty well past the point where timeouts would be effective, however he is ripe for confining him to his room. It may be hard at first, but it won't take very many room confinements before he learns that he can scream to his heart's content and it still won't get him what he wants. I let my boy have tantrums in his room and I wait (and do breathing exercises to stay calm) until he calms down before I talk to him. His room is safe, and he is old enough to be in there alone for ten or fifteen minutes...or longer, if that's what it takes.
Room confinement is a means to help diffuse emotional tension so that a child can listen well enough to have his behavior corrected, and a parent is calm enough to explain the difference between correct and incorrect behavior. It is not a punishment. Once everybody has calmed down, I ask my son how he feels, and why he feels that way. Then I explain that his tantrum was disruptive and not nice, but I understand how he feels. Then I give an example of proper behavior, usually couched as an example of how I behave when I feel that way. Finally, I have him apologize and then we move on without harping on it.
So once you have a plan for coping with his tantrums and your anger, you should be able to reduce those tantrums by making bedtime feel comfortable and safe for him. There are a lot of different reasons why your son might have objected to bedtime, but right now, for certain, he has a very negative association with bedtime, and you can change that quickly.
Start with a solid routine (if you don't already have one) and focus on making your child feel good about getting ready for sleep. For instance, you can praise him for the things he can do by himself, and compliment his cooperation for the things he needs help with. Also, remember that bedtime is a time when your child's fears and anxieties can surface easily because he is tired and not well trained at controlling his emotions. It is a good time to bond with him by encouraging him to talk about his feelings (when he is still in control of them), and by showing your respect for his attempts to act appropriately, even if he is not always successful. Everything takes practice, so take notice when he is exerting emotional control and encourage him when you know how difficult it can be. These small things will increase his trust in you, and he will be less likely to have a tantrum in order to get you to pay attention to how he feels because he will come to believe that you actually care about his emotions and will listen to him if he wants to tell you that he is upset.
My specific recommendations to help with bedtime:
- Give him notice. "When this show is done, it will be bedtime."
- Make a firm statement when it is time. "OK, the show is done. It's time to turn off the TV. Let's go to bed."
- Bribery is OK if you offer it before the tantrums start, but do not negotiate. "I'm bringing a little dessert than you can have after you put your jammies on."
- Praise the little things early on. "Wow! You got your jammies on so fast!"
- Stick to a clear routine. Predictability (consistency) will help cancel negative associations. "What do we do next? Brush our teeth!"
- Have a night-night ritual. My son always asks me what I want to dream about.