I found it difficult to locate any peer reviewed articles that attempted to define this concept, but Whittingham and Douglas (2014) state on p4 that
[B]abies are thought to easily become overtired if parents miss early tired cues, and an overtired baby is thought to have greater difficulty in falling asleep.
In my view this matches fairly closely how the term seems to be used informally in online discussions about parenting. The idea is that if you miss the opportunity to put a baby to sleep when they're mildly tired, they can get "overtired" and then it will be harder to put them to sleep if you had tried earlier. This is sometimes contrasted with adult sleep, where it gets easier to fall asleep the more tired you are.
Whittingham and Douglas seem to state that overtiredness doesn't really exist.
In fact, with increased tiredness comes a greater tendency to fall asleep—the very definition of sleepiness (high homeostatic sleep pressure) is that the individual, given opportunity, falls asleep.
I'm wondering firstly whether overtiredness exists at all, and if it does exist, at what ages does it operate?
Whittingham, K., & Douglas, P. (2014). Optimizing parent–infant sleep from birth to 6 months: a new paradigm. Infant mental health journal, 35(6), 614-623.