I think that the "As a kid you change your mind all the time. Once the neighbours are annoying and next day they're best friends" point has merit. Same for, " kids at this age change fast and this annoying little boy can become a friend for life."
However, I gather it hasn't helped enough for you to accept either answer or use that advice.
So my take on it in addition to the above:
You son has the opportunity to learn how to quietly and calmly insist that he is his own person and that under no circumstances is he going to be "mean to siblings (stealing toys), stamping on toys, jumping about with no regard to other people". He can tell this fellow that if they are going to hang out Neighbour has to grow up a little or Son is not interested. He doesn't have to be mean or rude, "I don't like this idea; I don't like this sort of game. I'll see you another time." If pressed, "I think that's kid stuff, and I'd rather..."
I'd help your son brainstorm some ideas for acceptable activities that your son would be willing to do with Neighbour as well as a list of things he won't do. You could role play conversations so that Son can learn to direct the conversation and respectfully decline bad ideas. Learning to stand our ground respectfully is a gift.
Standing up for ourselves is a communication skill that takes practice to master, because we want to do it without sounding defensive or disrespectful. This is a skill he will draw upon for his entire life. The goal is to demonstrate we are confident about ourselves, and also let people know that they cannot easily take advantage of us. "Just say, 'No'." (The basic rule for just saying 'no', is that you make no excuses and give no reason. That way, no one can help you fix the reason so that it can happen.)
As to your relationship with the neighbours: They know as well as you do that Son doesn't really enjoy their son's company -- they probably see way more of this negative behaviour that Son does. I see no reason for you to address it specifically unless your relationship is extremely 'mature'. If they ever ask you tell the simple unvarnished truth, "Son prefers less boisterous activities." It isn't very direct and so preserves your neighbourly relationship, but it isn't a lie either.
When I was a child, I was not at all like my brother. He was a sports-fiend and very active all the time. He was only a year younger and he was constantly beating me up -- albeit he thought it was play. We'd often end up in real fights. The fastest way to stop us was for both of us to get into trouble. Suddenly we were a united front. It might be a thought to allow them to create a united front, though how that might come about is beyond my crystal ball.
