You can get by with less than perfect. But you've already experienced a situation were you had to choose the "wrong" word. If that's rare then I don't see a problem.
However, since you say your level is around 3rd grade, I think that your son will catch up to your level quickly, and then it will become difficult for you. After all, as a parent and teacher you need to stay ahead. If you want to really only speak Mandarin to him although you don't feel sufficiently fluent, then I'd answer your title question as a yes that would be bad because you'll be running out of vocabulary too fast.
Even so: you've then taught your boy Mandarin up to a 3rd grade level, which is cool, and a good start into anything more later on -- if you can keep it maintained.
I've stated beforestated before that you've got to know all the words you're ever going to need, or else you'll be forced to fall back into another (or a simpler) language. I don't personally think that teaching a second language would work unless you're 97% fluent yourself, but your situation might be different, so don't feel put down.