I've seen this a lot, both in my kids and in their friends. My youngest went through this at the start of the year, and got a ton of help from his really excellent teacher on this - maybe that would be helpful here too. All of the below is from that experience and from discussions with teachers on the subject - I'm not an education expert nor a child psychologist, but I am a very involved parent who usually talks to the teachers much more than average.
In particular, from what I've been told when discussing this with teachers, it helps to recognize that not all of the skills you learn in school are "reading writing arithmetic" - in fact, in elementary school it's more important to learn the meta skills. "How to learn", "How to take an exam", that sort of thing. Once you learn those, you are set up for more success later.
For my child, the teacher recognized early on that he was far ahead material-wise; he's in third grade, just turned 9, and tests a few grades ahead in terms of material. But as you describe with your son, he was making skills mistakes, disorganized, and not good at explaining himself, along with poor handwriting for grade level.
As such, she recognized that she needed to change the focus. In his work, he received feedback based on how well he explained himself, on how well organized the answer was, and in particular on handwriting. The concentration seemed to come along with this - I'm not sure if she specifically worked on it or not, but I suspect the focus on organization and explaining the answers helped that a lot. Work with the teacher on this, and ask them if they can work with your son on these elements - and also explain to him why they're important.
I found that he would check his work and sometimes correct mistakes during this process of explaining, and this might be the first place to start for your child. If he's finishing first always, perhaps he should be going back and re-doing the problems. Or, if he's just providing answers, he should add some explanation for how he gets to the answer in the exam (if that's not already asked). Perhaps also explain how to check answers by reversing the problem - especially if he's making mistakes on things like addition/subtraction, those are easy to do (23+14=37; 37-14=23) or multiplication/division (9*5=45; 45/9=5).
For very bright kids, it's pretty common to just write down the answers to the problems they know are easy, and not work them out. That however leads to these silly mistakes. 19+43 = 52. Right? Of course, it's not... but it's easy to forget that carried ten. Explaining how they do the math can help that, because either they show the carried 1 on the paper, or if they do it like I do in my head (43+20-1) they can write that down also - and maybe if they do it wrong (say 43+20=53-1=52) they might spot it on paper when they don't spot it in their head.
Really anything he can do to slow himself down a bit will help - and also perhaps a talk about if he's feeling stress from the exam might also be worth having. Does he have pressure to succeed on these exams, either internally or externally? Does he feel like his self worth is defined by getting a good grade on the exam? That can be tough for a 3rd or 4th grader, and it isn't necessarily anyones' fault - though as a parent it's always worth considering whether you're contributing to that positively or negatively. I catch myself adding pressure sometimes to my kids, who frankly are doing far better in school than I have any right to complain about - but when you're "smart" you feel extra pressure to get even better results, and when your kids are "smart" you think they ought to get better grades, also... Perhaps if you talk to him about it, and see if he does have too much stress to perform well, it might help you find out if that's the issue.
Some relevant reading, perhaps: on test anxiety, on gifted students testing poorly.