There are a number of factors that contribute to your ultimate decision to keep or dispose of the milk, but an hour or two on the counter at room temperature will not by itself spoil the milk or make it dangerous or harmful to your baby. There are a number of components in breast milk that inhibit pathogenic activity.
La Leche League International gives 6 hours as an acceptable time at room temperature, with 4 hours being the ideal (http://www.lalecheleague.org/faq/milkstorage.html).
EDIT:
With previously refrigerated milk, I would err on the shorter end of that range, and would consider the age of the milk prior to being left out, but also note that it was cold when left out, and it may not have even reached room temperature during the time it was on the counter.
Personally, I would not keep that much of a supply out of the freezer, but only enough for the next feeding. I know that isn't always possible, but if it is, I recommend it.
What we do with our supply is to freeze expressed milk in bags until needed, at which point it is thawed/warmed in warm water. It takes a few minutes more minutes, but that is mostly just waiting time.