I don't see any reason it shouldn't work to help give your child the basics. Having taught ESL, having a half hour to an hour of English time is basically what I did with young children to get them learning the language.
If you try to speak anything more than simple sentences and vocabulary you'll just leave him confused and frustrated, keeping it simple, playing with toys, reading a story, watching an English cartoon or kids show is the way to go. Make it fun, lots of smiling, laughing, throw in some movement with simple songs and games, make faces while telling him what your nose, mouth, lips, hands, etc are, will have him picking it up really quickly.
Get your wife involved to. Even if she doesn't speak English well, when she's feeding him have her say the name of the food and dishes in English. Ask him if it tastes good and act out what it all means. Whoever washes him, name each body part as you wash it and tickle him a little. Play games with him by tricking him using English and Hebrew together. Keep it varied and interesting.
This is what I did with my daughter while I lived in China and was about the only English person who spoke to her. When we came to Canada she was maybe six months behind other toddlers and had a bit of an accent.