Wizard of OZ (WOZ) is a prototyping method that uses a human `wizard' to mimic the functions of a prospective system. Although the WOZ technique is widely used in HCI design, surprisingly little research has been done on how wizard performance and consistency may effect the findings of a WOZ trial. In this paper we present a meta-analysis of three WOZ studies that shows how wizard behavior can vary significantly, potentially influencing the user experience of test-participants. In text-based interactions, for example, wizards may be required to send several consecutive utterances before receiving any feedback. In order to deliver a consistent experience they then need to give test participants enough time for reading and processing an utterance before sending the next utterance. In this study we examined this aspect in some more detail and found that these time-windows can vary significantly within as well as between studies. By contrast, a similar experimental setup simulating speech-based system output did not suffer from this type of inconsistency. Our analysis highlights one of the challenges involved in using the WOZ method for designing novel interactive systems (both stationary and mobile), and suggests that additional support for wizards is needed in order to improve consistency.