Is it appropriate to install recording or other monitoring devices? Should the babysitter know about them?
It is not appropriate to install spycams for the babysitter
There have been recent legal cases in the US where monitoring devices were used to record the babysitter/estranged-father/grand-parents without their knowledge and the evidence gathered used in court. IIRC, they all backfired, badly.
If you are genuinely worried about your babysitter, get a different one.
The right babysitter for you will make you feel confident and relaxed in the knowledge that a capable person is looking after your children while you are out.
If your children are of a speaking age, ask them if they like the babysitter and what bedtime stories they read to them. If your children respond badly, you'll know.
While out, you can always call or SMS them for a status report, but we found the best method is to sometimes come back a bit early, come into the house very quietly and see what they're up to. If they're watching TV quietly or reading a book, no worries. Use your judgement.
If you are unsure about someone and can't get references, I would first try them out with you at your home. Maybe you can work from home or do some busy-work around the house to get a feel for them a few times. I would definitely go find references. I got many of my babysitting gigs from word-of-mouth.
Check out your state laws first. I know in NJ only one of the parties needs to know they are being recorded (in this case, you) but if YOU are not part of that recording, it becomes illegal. So you couldn't record a snippet of yourself and then leave a recording on to just tape the sitter.
I think that it's absolutely appropriate to know exactly what's going on with your kids. However, monitoring babysitter without her consent may be illegal in your country. That's easily solvable: get her consent! First, describe politely that you want to monitor your children (or just your house). You can name your reasons if you wish, like safety, security, etc. For this you are installing such and such devices. During the hours she's babysitting this may also record her and you'd like to ask if it's OK for her to be recorded. If she's not, don't start monitoring. Instead start looking for another babysitter. Interviewing a candidates you can mention that you have child monitoring system and ask whether they are fine with it recording them as well.
For legal reasons (or just to be sure), you may also need babysitter written consent: write a page describing the monitoring and ask her to read & sign. You may want to get a legal advice on the subject from your attorney to be sure.
BTW, just knowing there is some kind of monitoring may improve babysitter behavior, but it's better to avoid people whose behavior changes greatly. You don't want her to play angel in from of the camera and shout on/beat your child when out of view.