Will the baby be born after you move?
Think of the following: the noise to which a fetus/unborn baby is exposed to constantly while in the mother's belly is is far more than you can imagine. Accoring to this source (found on a quick research),
"ambient sound a baby hears in the womb - mainly blood running through your blood vessels and the movement of your stomach and intestines - actually reaches the level of about 90 decibels (about the level of background noise in an apartment next to an elevated train). While his developing ears can take those internal noise levels, exposure to very loud external noises can endanger an unborn baby's hearing."
This study aiming at measuring intrauterine sound levels just before and during birth states the following in the abstract:
We placed an electrically isolated microphone in the uterus of nine term gravid volunteers after amniorrhexis. Baseline levels of intrauterine sound were 72 to 88 db. Transabdominal vibroacoustic stimulation with an artificial larynx produced peak mixed frequency sound levels of 91 to 111 db.
This (non-scientific) website summarizes:
Instead of the womb being the quiet place scientists once assumed, it is actually awash in sounds, particularly the whooshing of your blood and digestive system, the thumping of your heart and your voice, which sounds louder than it would transmitted through the air since it reverberates through the bones and fluids in your body.
Considering therefore the sound level in the womb, the newborn is already used to noises, it is often not recommended to tiptoe around your newborn -- they are already used to noise from the womb, and you would only be making life harder for yourself.
That being said, I actually went through exactly what you describe to be your plan. My daughter was born in November, and starting with February the following year the inner garden and the underground garage of our apartment building underwent major renovation, lasting 10 full months. It was never a problem for the baby! The only negative reactions to noise were when some neighbour was drilling into their walls (that still happens, she's 2.5 y.o. now). Never to the noise outside. I, on the other hand, had a hard time dealing with it and appreciated the noise-free weekends. During the week I tried to hang to every opportunity to leave the house with the baby in order to get away from it, which was both good (meeting other friends with babies, offering my baby different stimuli, etc), and bad (I was too tired some days but still felt like I need to escape the drilling, or I couldn't rest properly when the baby was sleeping). I was however careful to cover her ears as well as I could when walking by the construction site if at that moment the noise was considerably loud.
To answer your question in short: there is very likely no damage from outside drilling/construction site noises for infants. Annoying to adults, yes, but not damaging to anyone