(I am a CISV volunteer, but will be objective in my response)
As mentioned by the others, CISV is a non-sectarian UNESCO partner non-profit organisation. I personally only joined as an adult volunteer in 2013, and can understand how it may initially seem like a sect to outsiders (if fact, this is something we might joke about ourselves, sometimes) - we cultivate a very open and fun culture and that aims to build a comfortable environment for children to learn and share in, and this is pretty much uniform across all chapters throughout the world.
I haven't been involved in an "Interchange," but I joined as an adult leader to lead a delegation of 4 eleven-year olds to a "Village" in Portugal, and have since also volunteered to lead "Youth Meeting" delegations to Belgium and Portugal, and even staffed/directed "Step Up" and "Village" programs in France, Sweden and Norway. These programs are very well-structured with clear learning goals behind all activities, adult training and risk management are always a top priority, and programs are constantly being evaluated and incrementally improved to adapt to the changing times.
As for your concern regarding the apparent involvement of mainly "upper-class, internationally-minded" families, it has been mentioned here that there is a certain investment that comes with participation (primarily membership fees to maintain local chapter operations and travel expenses to/from program sites), but is also a matter of greater geographic and economic situations - for example, mostly only children from very rich families in Brazil can afford to participate, while participation is well within reach for normal middle-class families in Northern Europe (which might explain the remarkable popularity of CISV across the Nordics).
In the end, it is in the children that families and volunteers such as myself invest their time and resources. Lessons learned and connections made within these programs can last a lifetime, and we provide a venue where children can be themselves and gain greater awareness of the world, in the hopes that this guides their future decisions and attitudes toward constructive and responsible actions. I personally understand that this might not be the best or most feasible option for all families, and I am not connected with your local chapter to speak on their behalf, but I hope my response has been able to help in some way.