It sounds like he may need help with his math, not encouragement to try harder. Just telling him to try harder isn't going to help if he doesn't understand it, so get him some help:
- Talk to his teacher. Explain how he is having trouble with the harder problems, and is getting frustrated because he really wants to solve them. A teacher's usually glad to help a student who wants to learn
- Work with him. If you or someone close to you is good at math sit with him and help solve the problem. He needs help, not encouragement
- Teach him to find help with math on the web. There's loads of math tutoring sites like Math.com that offer free help. It's not important that he solves the problems on his own all the time, learning how to help himself is empowering and good for self-confidence. As long as he learns the techniques he needs to do well on tests who cares how he learns?