I wouldn't stop when he says that. I'd simply tell him I will listen to him once he has heard me. If he refuses to stop what he is doing, I'd pick him up & remove him as he is young enough for that. If it were my child, I'd stop saying "listen to me" as it's obviously not working at this point & he is missing the point.
Think of it this way. Saying "Listen to me" is demanding respect. Going over & crouching next to them to speak is commanding respect. If you have to, then move on to picking up a child that is not listening & removing them to a less stimulating situation so that they understand that you do mean to make sure you are heard. If they head you out respectfully, then they may say their bit after you are done.
I've watched lots of kids in this age range & this is what I have come to believe. The child has to believe you will act. This does not mean spank or yell or take things away, simply that you will stop what you are doing & cross a room to address it immediately. If you leave distance & access to a 3 year old, they will often ignore you until you are right there, at their level & dealing with it. So if I start watching a child I don't normally have, the 1st week I say nothing corrective or positive from a distance. I go over to where they are & crouch down & say "Wow I love that picture you made" or "I think we need to put he blocks away since you are unable to play nicely with them". For some kids I assess that I should do this longer than the 1st week, it depends how many bad habits have set in before I started to watch them.
I also go over rules with them. I ask them what rules they think we should have too. You likely have to prompt them. At 3 a rule that is totally fair is that if you smear your food, you wipe it all up. If we were eating & ca child was smearing, I wouldn't bother to ask them to stop. I'd go over & say "Well it looks like you are all done eating, let's clean this up" and that is what we would do, together. I am kind, patient, etc, but you make the mess, you clean the mess & clearly you are not very hungry or you would eat the food versus smear it.
If he is being wild with a book, I'd walk over & say "Well it looks like you are too wound up for books right now as you aren't treating that right. Let's put that back & find you a more active toy, or go outside" (or whatever options you have).
The point is, children at this age are capable of clearing up their own mistakes AND they are also in need of direction often. So if he isn't playing well with books, he very well may need direction in what a better choice is in such a case. I do not think it is unusual for a 3 year old to be unable to reign themselves in simply by you telling them to stop doing something. In fact, they may stop, but are then likely to make an equally bad choice on what to replace it with. They do not read their own bodies well (such as I feel super energetic, I should go jump on my trampoline), they only know what interests them in this very moment. I have a wild wild 3 year old right now. I make sure I have good options to help her deal with her energy level. She has indoor toys that help get some energy out as well as outside very active play, etc. It's important. When she is overly energetic, she isn't being bad, she is being a ball of energy. She has to be told what she can do with that energy & how to direct it positively. I wish I could turn it into cleaning & dusting, but alas I can settle for 30mins of bouncing on her indoor trampoline & then I get a kid that is much more ready to sit & read a book than to rip it to shreds & stuff the pages into the sofa.
Other tips for kids in this age range & very active personalities are things like setting up a "no fail" environment. That means taking the responsibility to remove trouble spots from their most common areas. It is merely a frustration to have to say the same thing to the same child all the time. It makes you less patient and it does the same to them. And try to avoid saying no. Children can learn to be patient, wait, etc, but it's much easier if you drop the no. You replace it with things like "Yes we can go to the park tomorrow. Today we have other things we need to do". So see...you aren't saying yes we can do what you want right now, but saying no is unnecessary most times & it's less frustrating to get a yes, even if it's a "yes later". Other ways you avoid no are things like, "Well would rather do that than insert whatever other thing they might like as an option that is easier for you", "Here, you can have this thing" (like say they are messing with the remote, replace it with some other button pushing item that is okay, like a calculator), etc, etc. And offer as many options in a day as you can, but keep them simple & not bossy. So if you have to leave you can say "Well it's time to go, do you want to put your shoes on first or your jacket?" And my final tip is to give them a head's up on all things you do. So I stay about 3 steps ahead of where we are. Say we are at a playground. I will say "Okay we have about 30mins here, then we have to make a stop at the grocery store, and go home and start dinner". When we have about 5 mins left I say that all again, saying we only have 5mins, then I give a 2min warning & 1 min & then have them say goodbye to the playground. As we are getting in the car I remind them that we are heading to the store, going to make dinner & then after dinner they will get a bath. For small kids our way of being often comes across rather random & bossy. We are constantly telling them where to go, when & right now & without warning, etc. It leads to conflicts that are unnecessary often because it's merely a normal response to be annoyed that you were doing something "really important" (in the middle of play) and now someone declares it's over & you are off to some random place you didn't even know was happening today.
Soooo that is all my tips. I hope any of it helps.