Beast Academy has books and online materials for teaching math to kids 8-13: https://beastacademy.com/ .
From what you wrote, your 5-year-old brother could be at the 9-year-old level or higher. For our child, Beast Academy worked really well. Their program is very fun and engaging for the child, and teaches math concepts in a systematic way starting from simple to pretty advanced topics. I recommend to buy both the books and the online version. These are well worth the money, considering in comparison the costs of private lessons, tutoring, etc. The children can master most everything on their own, and need adult help only with the most advanced problems.
If you prefer online-only materials, the more well-known Khan Academy could also a good choice: https://www.khanacademy.org/math/k-8-grades .
In terms of the bigger picture, as your brother enters school, I recommend also considering programs for gifted children in your public school system. Such programs tend to have teachers specifically trained to teach gifted kids, including the so called twice exceptional children, which could be the case with your brother. Not all states and municipalities have those, unfortunately, which then leaves only private options. In any case, extracurricular tutoring in math could be a great supplement to the school curriculum. It is always a good idea to keep math challenging to the child and provide opportunities for the so called deliberate practice, to keep them interested, engaged, and working hard developing their abilities.
SEE ALSO:
Jan Davidson, Bob Davidson, and Laura Vanderkam (2005) "Genius Denied". Simon & Schuster: https://www.amazon.com/Genius-Denied-Wasting-Brightest-Young/dp/0743254619/
RELATED:
https://parenting.stackexchange.com/a/39193/33055
