I'm not disabled myself, but my daughter has cerebral palsy, so I think about things from her point of view a lot.
My opinion is that you should give priority to someone waiting who needs a disabled stall, but don't worry about using it otherwise, whether you have small children with you or not. While I appreciate people who go out of their way to help my daughter, I don't consider it horrible to have to wait a little bit like anyone else sometimes. In fact, we are accustomed to patiently waiting for accommodations, such as at restaurants we often have a longer wait than others in order to get a suitable table.
That's not to say other people won't be "shocked and appalled" that you forced a disabled person who came in after you to wait a few minutes, but in my experience with my daughter and other acquaintances, the person himself probably won't mind at all, and their opinion is more important. Not to mention the chances of making a disabled person wait are very small.