I've been there before. My son was over a year old before he started sleeping through the night. Here are my suggestions:
- Put him to bed earlier - like 7pm. It sounds counterintuitive, but sleep quality goes down when you're overtired, leaving you prone to waking more (and the same is true for children). As the saying goes "sleep begets sleep."
- Do not stop his nap if he's down to one. If he's still at two, try consolidating. Children his age still need a mid-day nap, and the nap should be at least an hour (some children can do well on 45 minutes, but generally speaking, an hour is needed to get a ful cycle). Again, sleep begets sleep.
- Make sure he's eating enough. It's hard to sleep when you're hungry. We generally give our son a granola bar shortly before bed, and of course, make sure he's eating enough throughout the day.
- Keep a cup of water in his room. I know the air in my house tends to be dry, so our son has a sippy cup with water in it for when he's thirsty. He gets up, drinks some, and goes back to bed.
- Make sure his room is comfortable. If he uses a blanket, make sure his pajamas aren't too warm. If his room has a ceiling fan, turn it on to keep the air circulating. If the air is too dry, put in a humidifier.
- Get him a new mattress. This one will depend entirely on your child, and of course make sure to still keep the sleep environment safe, but I found that my son slept a lot better when we switched him to a twin mattress (his bed is now just a mattress on the floor) from his old infant/toddler mattress. He simply couldn't sleep on a mattress as firm as the infant mattress. Switching to a softer mattress worked wonders for us.
- Follow his circadian rhythm. We all have our own sleep schedules, but they generally follow a particular pattern. For young children, it's easiest to start with "the 5 hour rule." That is, awake for 5 hours, then sleep. So, if your son gets up at 7am, put him down for a nap at noon. Assuming he takes a 2 hour nap, then put him to bed at 7pm. The key is to find his natural patterns and make a schedule out of it, putting him to bed during those key "windows" when getting him to sleep and keeping him that way will be easiest.
- Pick a "sleep training" method and stick to it. It doesn't have to be exactly the "leave them and don't return until morning" or the "Ferber method", just as long as you're consistent about it. We're at the point now that, for the most part, we can lay him down and walk out of the room. For the nights that aren't that easy, we make sure his needs are taken care of (food, water, dry diaper, etc), and if he's doing it just to be stubborn/defiant, we let him have his tantrum. When it sounds like he's starting to settle down, we go in and lay him back down and tuck him in. One of the first things I did, too, was sit by his bed and not allow him to climb out (I don't like locking a small child in their room if I don't have to). For my son, at least, it sends the same message - it's time for bed, and no amount of fighting is going to change that, but does so without locking him in his room alone (which, I think, helps convey that I'm not abandoning or neglecting him, but that I'm enforcing a rule). It took a good half hour, but I only needed to do it once to get him to stop fighting bedtime all of the time.
- Encourage him to choose to go to bed. Bed time shouldn't be a negative thing, if you can help it. Make sure he knows that it's okay to ask to go to bed early. For our son, it helps to make him feel like a "big boy" if he does such things, and explain things like "big boys know when they need to go to bed early" (telling him about what big boys do is a great motivator for our son).
- Have a pre-bedtime routine. It doesn't have to be elaborate, just consistent. For us, it's my husband starting a sort of "countdown," starting about 6:30. He'll tell our son things like, "it's almost time for bed," and "you have 5 minutes left before bed." If our son hasn't had a snack recently, then we'll get out a granola bar or some yogurt and give it to him. After he's done, hubby has him say good night to me and walks him back to his room, where he changes him and puts him to bed.
- Do you have music or white noise of some sort playing in his room? If not, that might be worth trying. If his life is busy and noisy during the day, it might be too quite for him to be comfortable. Some soft music or some other white noise might help (conversely, if you do have music going, it might be the cause, so it might be worth tring a few nights without).
- Have his two-year molars come in yet? If not, that might be one of the things waking him and keeping him awake. It might not seem like it during the day, but he's likely too distracted to notice or care about the pain, but once night comes and the distractions are gone, there's nothing left but that pain.
- Talk to his pediatrician. You mention that he wakes distressed. It's possible that he's having night terrors, or may be waking up, disoriented, from a sleepwalking episode. While some of both are fairly common at his age, the frequency may be a sign of an underlying medical condition. Your child's doctor would be able to help you determine if this is the case and refer you to a pediatric sleep specialist if necessary.
- Try to also teach him sign language, especially if he's not actually talking, or his vocabulary is still limited. Being pre-verbal is particularly frustrating, because toddlers are really smart, but they can't communicate things in a way people understand. Knowing some signs can make life a lot easier on both of you, because then he can communicate what he needs/wants during the night wakings (and no, it won't hinder his verbal skills).