Verification of Results

After optimizing the rules and parameters of a trading system to obtain good behavior on the development or in-sample data, but before risking any real money, it is essential to verify the system’s performance in some manner. Verification of system performance is important because it gives the trader a chance to veto faime and embrace success: Systems that fail the test of verification can be discarded, ones that pass can be traded with confidence. Verification is the single most critical step on the road to success with optimization or, in fact, with any other method of discovering a trading model that really works.

To ensure success, verify any trading solution using out-of-sample tests or inferential statistics, preferably both. Discard any solution that fails to be profitable in an out-of-sample test: It is likely to fail again when the rubber hits the road. Compute inferential statistics on all tests, both in-sample and out-of-sample. These statistics reveal the probability that the performance observed in a sample reflects something real that will hold up in other samples and in real-time trading. Inferential statistics work by making probability inferences based on the distribution of profitability in a system’s trades or returns. Be sure to use statistics that are corrected for multiple tests when analyzing in-sample optimization results. Out-of-sample tests should be analyzed with standard, uncorrected statistics. Such statistics appear in some of the performance reports that are displayed in the chapter on simulators. The use of statistics to evaluate trading systems is covered in depth in the following chapter. Develop a working knowledge of statistics; it will make you a better trader.

Some suggest checking a model for sensitivity to small changes in parameter values. A model highly tolerant of such changes is more “robust” than a model not as tolerant, it is said. Do not pay too much attention to these claims. In truth, parameter tolerance cannot be relied upon as a gauge of model robustness. Many extremely robust models are highly sensitive to the values of certain parameters. The only true arbiters of system robustness are statistical and, especially, out-ofsample tests.