No Verification

One of the better ways to get into trouble is by failing to verify model performance using out-of-sample tests or inferential statistics. Without such tests, the spurious solutions resulting from small samples and large parameter sets, not to mention other less obvious causes, will go undetected. The trading system that appears to be ideal on the development sample will be put “on-line,” and devastating losses will follow. Developing systems without subjecting them to out-of-sample and statistical tests is like flying blind, without a safety belt, in an uninspected aircraft.