It sounds like he may need help with his math, not encouragement to try harder. Just telling him to try harder isn't going to help if he doesn't understand it, so get him some help:
- Talk to his teacher. Explain how he is having trouble with the harder problems, and is getting frustrated because he really wants to solve them. A teacher's usually glad to help a student who wants to learn
- Work with him. If you or someone close to you is good at math sit with him and help solve the problem. He needs help, not encouragement
- Teach him to find help with math on the web. There's loads of math tutoring sites like Math.com that offer free help. It's not important that he solves the problems on his own all the time, learning how to help himself is empowering and good for self-confidence. As long as he learns the techniques he needs to do well on tests who cares how he learns?
- Try to find practical applications. My mother had me practice my math by balancing her checkbook and that taught me more than one lesson
As to how to keep him from quitting in general, so much of it is about self confidence, building his self-confidence is key. If he learns that hard work brings success and reward then you'll get more hard work. So praise and hard work and perseverance. Start easier, than give him more and more challenging things to do, and encourage him to think of his own challenges.