If you have a business, you probably balance your checkbook monthly or more frequently. This is a way of making sure you’re in sync with your bank. The same should be true for any mystery shopping program.
You should check on a regular basis to make sure your observations are in sync with your business. More frequently if you’re trying to make changes to an existing organization. Keep a core group of questions/observations the same, but mix-up or add to each of the basics.
Conversely, you would never want to start all over from scratch. Your employees may interpret a massive change like this as your disinterest in the program, or worse you could loose credibility.