Actually, yes. It is possible he is purposefully riling you up. Not, perhaps for any specific reason he is consciously aware of. It is also possible that he simply gets distracted and doesn't know how to cope with your anger and unwittingly makes you more angry with his reactions.
If he is purposefully riling you, he may do it because (even if he does get a lot of attention) he may feel he needs more attention. He may do it because of a latent anger about something or because something thrills him about getting you upset. There may be underlying issues I cannot even begin to guess at. One possibility is that he is a smart kid not motivated by standard rewards and punishments who wants to do things (like wash his hands) on his own terms.
After teaching for ten years (including in a specialized school for kids with emotional and behavioral challenges), I've concluded that some kids are only intrinsically motivated. These kids also like to be in control of their own destinies (don't you like to be in control of yourself?) Recent research has confirmed that for many kids rewards AND punishments are not really the motivators many parents believe them to be. The methods used for teaching and disciplining these kinds of kids, convieniently work really well with other kids too and teach all kids life skills about self control, problem solving and decision making that are important to their well being.
Answering a question about how to control your own anger is probably best suited to another forum, but in terms of the parenting aspect of things, you will be well-served to avoid getting highly emotive about most things related to your child and his foibles. He may just need a listener.
Offering up Choices
To begin, it may help to offer him choices when you can. I understand choices are not always possible (nor, really should they be) but the more of them you can offer, the better. These are not choices like wash your hands or you can't . . . because that isn't a real choice, it is a threat. You might choose two ways he can wash his hands that are both equally comfortable for you. "Would you like to wash your hands at the kitchen sink or in the bathroom?" by doing this, you are telling him what needs to be done, but also offering up a choice and allowing him some say in how he goes about doing the hand washing. This offers him a sense of control and/or dignity he may subconsciously need and is not getting. The more you can offer such choices in general, the more this need will be met and the less you will find yourself in a power struggle.
When you offer choices often, when a time arises that you can't offer a choice, you are free to say, "Dude, I let you decide a lot of things around here. Once in awhile you've gotta just go with it and trust me on some things - this is one of those times."
Seeking Understanding
He is a kid and needs to be understood to feel loved and accepted. You are a parent and need to feel you understand him so you can do a good job in guiding him. Why not ask questions?
"Hey (name here), do you remember what you are supposed to be doing?"
"no"
"Oh. Well, you need to wash your hands so we can (go shopping, go to your friends, so you can play . . .) and you seem to have chosen to wash your hands here in the bathroom."
OR
"Yes"
"Hmm. . . Can you tell me why you aren't washing them? Is there something I can help with?"
If he is speaking calmly, listen calmly. If he speaks in anger, take a deep breath and listen calmly. Show him that no matter what, you love and accept him - I know you just also need to teach him a few things along the way, but show him love and acceptance first.
Ask clarifying questions and paraphrase him. Paraphrasing you will let him know you understand his response fully.
Then, you can tell him your thoughts. Tell him whatever it is, calmly and start with "I."
"I need you to get your hands washed so we can move on with our day. I know you don't want to wash them now, but that is not a choice. Your hands need washing. I will expect them washed in two minutes."
If, after all that he still doesn't wash his hands, do it for him. Hold his hands (not with anger or emotion) just hold his hands and go through the motions of washing his hands. If this angers him, calmly say, "Well, if you show me you can handle washing your hands on your own next time, then, you won't have to let me wash them for you."
For most things, natural consequences like this are much more likely to help better than you yelling at him, punishing him, or offering rewards for a job finished (bribing him).
The thing that is super important with this method is that you Remain virtually emotion free. That is, except when you are expressing empathy. Consequences arise because they are the natural order of things. Consequences (good AND bad) do not arise because of your anger, frustration or even because of your happiness in his choices or pride in his successes.
If he makes a poor choice, feel badly with him about it, but don't jump in and rescue him from his own poor choice. Remember - it isn't really your problem so you have no reason to be angry about it. When he is behaving in a way that does pose a problem, tell him about your feelings, "I am frustrated because I have asked you to (whatever it is, wash hands for example) and you said you would do it and now you are not. Here is what I am going to do about that." Just make sure your tone and body language are both calm while you tell him about your feelings.
Don't forget to be there for him to talk about ideas that may have worked better for him when he does make bad choices. Together, brainstorm ideas about different choices he could use next time that will be more productive. It is a difficult thing to switch to and requires a lot of work but gets MUCH better results in the end.
You are also well within your rights to do some family counseling if you choose to. A little conversation with an expert around to do some of the conflict resolution and teach all of you skills that will help you understand each-other better is a good thing for anyone.