Great question. There are several different reasons one might want to limit TV time for kids, and understanding those reasons can help support informed decisions about when --- and how --- to let kids watch TV. There are three potential problems with TV time:
- TV replaces other activities that may be more valuable
- Some TV content may not be appropriate for kids
- There may be negative effects of TV exposure itself
This is an area of on-going research, and all three of these are complex issues, so nothing is cut and dried. I've pulled together a couple resources that might be useful for you, though.
TV replaces other activities that may be more valuable
Many of the bad effects of TV are related to what's not happening during TV watching rather than what is, as you suggest:
Or is it rather a problem simply because of what the children might otherwise be doing if not watching TV (e.g. reading, socializing, physical activity, and so on)?
For example, there is evidence that higher TV watching is associated with health problems such as obesity (citation), but that's likely just because it's a sedentary behavior, not because the TV exposure itself is harmful (i.e. a child who spent at time sitting quietly on the couch not watching anything would probably be at the same risk as a child who spent that time watching TV).
There are also studies showing an association between more TV time and delays in learning to read (citation). In this case, the probable reason for the association is that children who spend more time watching TV spend less time reading, so they're getting less practice and are therefore slower learning than their peers who spend more time reading and less time watching TV. Similarly, more TV time is associated with slower language development, but this relationship can be completely statistically explained (a mediation model) by the amount of language kids hear from adults (study); in other words, the reason kids who watch more TV learn language more slowly is because those kids are hearing less language from their caregivers.
So basically, the risk is that there are only so many hours in the day and the more time you spend watching TV the less is left for other activities. That's potentially a serious concern, because TV watching is an activity with very few positive features (very little physical movement, very little problem solving or reasoning, very little social interaction, etc.).
Some TV content may be harmful for kids to consume
There are also studies showing negative effects of TV content itself, such as possible aggressive behavior, poor body image, substance use, and poor school performance (here is a review, including citations for several supporting studies). In many of these studies, the issue is appropriate content --- kids are watching programs with violence, substance use, sexualization of women, etc. which can influence their thoughts, ideas, and values.
Parents being involved and aware of their children's TV consumption (and setting rules about what is or isn't okay to watch) can prevent many of these issues.
So what if you make sure your kid gets lots of time doing healthy activities, and you're careful about them not watching any inappropriate content? Is there still risk associated with TV time?
Negative effects of TV time itself
Even when watching child-appropriate content, there is still some evidence to suggest that more TV time is associated with attention problems later in life (citation). There are also immediate effects of high-energy TV such as cartoons, which make it harder for children to concentrate and reduce their ability to control impulses after watching them (citation). In that study, children were randomly assigned to watch either fast-paced cartoons (Spongebob Squarepants), educational TV (a PBS broadcast about a preschool aged boy), or to color. Right after, the children were measured on a variety of tasks designed to assess executive function skills such as attention, following instructions, and impulse control. Children who had just seen the cartoons showed impaired performance across the board. Children who had watched the slower-paced PBS show showed more or less normal performance, and children who had been coloring showed mostly normal performance with some tasks better than normal. This study illustrates that even among children's programming, some shows may sap cognitive resources while others don't.
So there's evidence that TV can impair cognitive functioning. Can TV still play an important role in education? From the studies that have been conducted on this topic, the answer appears to be, "Maybe for older kids, but not for infants and toddlers under 2." I've already compiled sources on this for my answer to another related question, so I'll just quote the relevant piece here:
This study tests toddlers' ability to learn new words
from conversations watched over video compared to in person. This study tests particular videos explicitly marketed to help infants
learn (Baby Einstein videos), showing that infants don't actually
learn new words from them without significant scaffolding from
caregivers. (Here's a review that covers several similar studies, if
you want to learn more, and another)
There have also been a few studies testing how well infants can learn
a foreign language from exposure to media (video recordings), the most
famous of which is probably this study showing infants don't learn
much about the sound structure of a foreign language unless they get
exposed to it live, in person.
Here is a great review article on evidence about infants' ability (or lack thereof) to learn from TV, including a precise discussion of the special circumstances under which infants do appear to be able to learn from TV exposure. It also contains citations for many more studies on this topic, so if you want to learn more that's a great place to start. Another study investigated circumstances under which 2-year-olds were able to imitate a new skill learned in a video, and found that they could do it under some circumstances but not others.
Older kids (>2yo) and adults definitely can learn things from TV, of course (I've learned plenty of cool things from TV myself). But infants and toddlers really don't, unless you provide a lot of social support for them, watching and discussing the content together (and, at that point, it's hard to say if the child is really learning from the TV program or just learning from you talking about the TV program).
As with any activity, there are both pros and cons --- I'm not suggesting that we should all have a no-TV-ever policy for kids. Watching TV is fun, and it can be a nice opportunity to relax and cuddle. But in order to make decisions you feel good about, it's important to be aware of the risks of TV exposure and the benefits.