While I am not (yet) in your situation, I've been reading a lot about it and it seems many kids do lose the minority language. While kids are very flexible at learning, they are very flexible at unlearning too.
Consider how well you want him to know Portuguese. Well enough to converse with the family is a much lower threshold than well enough to be able to write scholarly Portuguese, for example.
If you can expose him to Portuguese speaking kids, that would be ideal. Most often, if you a really a minority in your area (and not like Spanish speakers in many parts of the US), these will be his family (cousins, &c) who do not speak English very well. Have him spend holidays with the family if possible.
In your day to day interaction, they need to be talked to (not just exposed—TV or passive hearing is not enough—but interacted with) at least a few hours a week in a language or they will lose it. The amount of time can vary, but think on the order of at least 20% of their interaction should be in Portuguese for them to be truly fluent. Some families have rules like "only portuguese in the house" or something. The rules do not matter as such, but they might force you to talk enough Portuguese to go over that threshold.
I recommend Raising a Bilingual Child by Barbara Zurer Pearson where you can find scientific support for all of the above.