There's a small but serious risk of death or harm if the TV falls on the infant. Please make sure the TV has a good quality stand, and preferably is secured to the wall if possible.
About one child dies every two weeks in the US from falling TVs.
But your question is not about falling, it's about the cognitive stuff.
The American Association of Pediatrics recommends zero screen time before 2 years old. That's about the most restrictive credible recommendation I can find. But that recommendation is not asking you to prevent your child from ever seeing a screen before the age of 2. It's telling you that face to face time is better for the child than tv time; that tv time doesn't provide any benefits, and that excessive screentime might be mildly detrimental to language development.
So there isn't anything to worry about from occasional tv watching.
And the AAP is starting to look at this guidance to see if it's still the right advice.
However, it's a bit rude of Grandpa to ignore your wishes. Your request is reasonable, not outlandish, and they should be listening to you. I have no idea how you'd ask them to start paying more attention to you.