Research into attachment disorders like reactive attachment disorder show that inattentive primary caregivers or sudden changes of primary caregivers under age 3 or so can have permanent effects on a child's ability to bond with future caregivers, no matter how stable the relationship is after that point. It's something foster and adoptive parents are warned about and trained to handle.
Note that doesn't mean everyone with that kind of instability ends up having attachment issues. Nor does it mean you can't form a lasting bond with someone you met after age 3. Nor is a relatively detached relationship necessarily a result of an attachment disorder. It's just that age 0-3 is when the way you form attachments gets wired into your brain, and it can sometimes be hard to undo if you don't learn it then.
To put it another way, people with psychological attachment issues have problems bonding with everyone, not just their parents. If your brother has other close and lasting relationships with friends or his wife or his own children, it's unlikely the 3 years of being raised by grandparents damaged his ability to bond with your parents, and more likely his detachment is the result of ongoing factors.