I think the best way to deal with other children doing things you think are wrong is to confront them and ask them to stop or change their behavior in the same way that a nursery school teacher or other caretaker would.
I believe that all adults are responsible for the upbringing of all children. This principle is behind many written and unwritten rules, such as not crossing at a red light even in the absence of traffic when children are present who might immitate a behavior that could endanger them.
Taking on the perspective of a nursery school teacher is especially helpful if your own child is the victim. The best approach is to think of yourself as taking care of both children, as that will help you reduce your agitation and prevent you from doing something unlawful (such as pushing back the other child).
In your case
Often the other child is gone from your "care" (in the sense defined in the previous section) or the other child does not much care about your attempts to take part in his education. In that case, it is certainly helpful for your child to learn that the behavior of the other child was wrong in your eyes and that "we don't do things like this".
This is important because often children find fault in themselves for being bullied, and telling them that the other party was wrong and they behaved well takes that self-blame away.
Also, children learn from other children. It is a common observation that victims of bullying often become bullies themselves*. If your child sees that pushing other kids gets you want you want, they may decide that they'd rather push other children around than always be at the receiving end of that behavior. It is very important to learn how to deal with the emotions of being victimized without turning to retaliation or self-hate.
* Researchers differentiate between bullies, victims, and "bully/victims" or "bully-victims". Bully/victims are children who are both victims and bullies. In studies they typically make up around 6% or 7% of the sample (e.g. Haynie et al., 2001). In longitudinal studies, it becomes apparent that these bully/victims have usually been victims first and turned to bullying later. For example, Haltigan and Vaillancourt (2014) found that "for ... 6% of the participants, there was evidence that children started off as victims and then became bullies". In cyber-bullying among students the frequency of bully/victims is even higher: about 25% of all students are bully/victims, while only 5% are either bullies or victims but not both, Mishna et al. (2012) found.