What a difficult, painful and important issue. And congratulations on recognizing the long-term effects that the situation might have on your children.
Divorce can affect a child's relationship with their parents, and creates stresses which can interfere with their natural development. While divorce per se does not seem to negatively impact children in the long run, the factors most important seem to be the quality of the relationship between the parents, the quality of the relationship between the child and each parent separately, and the amount of contact between child and non-custodial parent. The negative effects of divorce are greatly mitigated when both parents maintain a positive relationship with each other. This seems rather intuitive, doesn't it? The problems continue to affect them into adulthood as well.
In terms of your son's comments, he may well not understand what he's saying to you. Reacting calmly without any hostility (as you seem to have done) is very important. Clearly you don't sound like someone who would go the tit-for-tat route. But above that, never criticize their mother (to them or in front of anyone where it might get back to your ex). It is not only bad for your children, but if they repeat anything they've heard to their mother, it is the children who will pay in the end.
I don't know your situation, but generally, what can be done? First, have you spoken about this with your wife? If not, why not? Remember, though, that showing empathy in your dealings with your ex and refusing to engage in conflict is not about your ex; it's about your kids.
Second, (with very good reason), you left your wife. The spouse that was left tends to be more bitter. What was done before you left? Did either of you receive counseling or did you have marital counseling? What effect did that have (obviously you are now divorced, but it would help to know if your wife responded at all to it.) Counseling together as divorced co-parents might result in decreased bitterness, reduce her negative comments/influence and might help her to see how damaging this is to children in the long term. If the last element is the only thing she gets out of counseling, it would be worth it.
Third, income of both parents drops dramatically after divorce. Keeping very current with alimony, and absorbing some extras (running your children to playdates, paying for lessons, etc.) might help. Not only might your children see you going out of your way to be nice to their mom, but she might actually be appreciative as well.
Also, it seems to be clear from the literature that any kind of therapy improves outcome in ~70% of cases. If your wife isn't interested in counseling, you can go by yourself to learn coping strategies, or have co-counseling with your children (yes, they do have interventions with children this young).
If all these measures seem premature to consider, then do what you're doing now: love them abundantly. Gently correct any misinformation they might repeat to you, as you did. Feeling words (that must be confusing, etc) are part of the language of love. Keep open communications with teachers (again, congratulations) to detect problems early.
Let them know they can talk to you about anything (and that it won't be brought up with their mom). Trust is part of love. Providing a safe emotional environment where they don't feel that they're in the middle of some war-torn area is wonderful for your kids.
You can get more advice here at Parenting, as well as from good books on the subject and good blogs (perhaps like this one about collaborative divorce, and this one about co-parenting).
Post-Divorce Family Relationships as Mediating Factors in the Consequences of Divorce for Children Hess et al. 2010
The Effects of Divorce and Marital Discord on Adult Children's Psychological Well-Being Amato et al. 2001
Children's Adjustment in Conflicted Marriage and Divorce: A Decade Review of Research Joan B. Kelly 2000
What works with children and adolescents?: a critical review of psychological interventions with children, adolescents and their families A. Carr 2013
Preventive Interventions for Children of Divorce: A Developmental Model for 5 and 6 Year Old Children Pedro-Carroll et al. 1997