To forewarn you, some of this is going to be my own views, and they seem to differ from yours.
Swearing isn't a problem by itself
If I stand in an empty room and shout obscenities and no one can hear, does it matter? No.
Offense is in the mind of the listener
If I say a word in your language which I don't understand, it is not offensive until you hear it and interpret it. Offense is a choice made by the listener to be offended.
Thus, offensive language is entirely context-dependent
The lesson to teach a child about swearing is that it is part of speaking for the benefit of those who are listening. So, we don't swear in front of young children because they won't understand that these are words that cause offense. We don't swear at teachers or parents unless we want to offend them and make them angry.
Not swearing is a reasonable choice if you don't know whether people will be offended or not
You can make it clear that you do not accept offensive language, and that you will assume they are trying to make you sad or angry if they swear in your presence. Further, you can point out that unless you know everyone in the room is fine with swearing, it is better not to swear because you might upset someone when you didn't mean to.
You can even suggest that the simplest thing is to never swear, then you don't have to worry about it.
A word is made of sounds
I have a real issue with just teaching children that all swearing is really bad. Swearing is just a bunch of sounds put together, and by themselves they are meaningless, not some sort of magical phrase. Further, it fails to teach children that the central point of language use is the context of the listener - that all language, the tone in which you say it, etc, is capable of causing joy or suffering in others, and that you have to think about the impact of what you're saying.
Frankly, swearing is the easiest way to learn that principle, so I would use it as an opportunity to teach them the concept rather than a rule they will become happy to break with their friends because it's forbidden.
Or, you know, teach them an over-simplified view of the world and wait for it to bite you when they're teenagers.
What to do in that social situation?
If you are in a situation where you feel your child is being influenced towards bad behaviour, then take them out of the situation. However, I would argue (given the above) that swearing itself isn't a particularly big problem, especially for a 9 year old who likely knows all the swear words anyway.
"Rough" children are not a problem, so avoid moving your son away from people just because of their social class. The problem is that you disapproved of their behaviour. However, in their context, it could be harmless (per the above discussion about context). That's why I've described my views on swearing, so that you can hopefully understand that you can explain why we don't use those words at home later. You don't need to be scared of your son hearing the words or even for the toddler. Nobody is harmed by hearing swear words. The key is for your son to learn the context, not for him to be shielded from some terrible utterance.