I've not been brave enough to try to teach my son German, my best non-native language, but our son is growing up as a bilingual Japanese-speaking child in the US.
In our case, we have the advantage that mom is a native speaker of Japanese, but we've observed that this is not enough; several parents we know have children 3-10 years old that simply refuse to speak Japanese, even though they can understand most of what mom or dad say in Japanese, simply because that's not what their friends are using.
However, mom is closely connected with other Japanese-speaking moms around our city, and this has reinforced the value of speaking Japanese, because so many friends of the same age as our son are also speaking Japanese. I'm convinced this is the single most important thing for retaining and improving the non-dominant language. It's absolutely no problem for our son to learn the dominant language where we live; my son speaks an odd mix of English and Japanese to me (he's only 2, so he's not completely code switching yet), but it does take some social reinforcement to make the non-dominant language seem worthwhile to a child.
If I were going to move back to Japan, the single most useful thing to help our son's English skills would be to develop a network of native English-speaking friends with children. I'd make sure we spent at least some time each week reinforcing those friendships with other kids and creating opportunities for them to play and eat together. I'd emphasize the same thing in the unlikely event that we moved to Germany; send the children to the local schools if practical, but make sure they have a mix of Japanese-speaking and English-speaking friends that they see frequently (and perhaps somewhat separately from their friends that are only speakers of one language).