Analytic Optimizers

Analysis (as in “real analysis” or “complex analysis”) is an extension of classical college calculus. Analytic optimizers involve the well-developed machinery of analysis, specifically differential calculus and the study of analytic functions, in the solution of practical problems. In some instances, analytic methods can yield a direct (noniterative) solution to an optimization problem. This happens to be the case for multiple regression, where solutions can be obtained with a few matrix calculations. In multiple regression, the goal is to find a set of regression weights that minimize the sum of the squared prediction errors. In other cases, iterative techniques must be used. The connection weights in a neural network, for example, cannot be directly determined. They must be estimated using an iterative procedure, such as back-propagation.

Many iterative techniques used to solve multivariate optimization problems (those involving several variables or parameters) employ some variation on the theme of steepest ascent. In its most basic form, optimization by steepest ascent works as follows: A point in the domain of the fitness function (that is, a set of parameter values) is chosen by some means. The gradient vector at that point is evaluated by computing the derivatives of the fitness function with respect to each of the variables or parameters; this defines the direction in ndimensional parameter space for which a fixed amount of movement will produce the greatest increase in fitness. A small step is taken up the hill in fitness space, along the direction of the gradient. The gradient is then recomputed at this new point, and another, perhaps smaller, step is taken. The process is repeated until convergence occurs.

A real-world implementation of steepest ascent optimization has to specify how the step size will be determined at each iteration, and how the direction defined by the gradient will be adjusted for better overall convergence of the optimization process. Naive implementations assume that there is an analytic fitness surface (one that can be approximated locally by a convergent power series) having hills that must be climbed. More sophisticated implementations go further, commonly assuming that the fitness function can be well approximated locally by a quadratic form. If a fitness function satisfies this assumption, then much faster

convergence to a solution can be achieved. However, when the fitness surface has many irregularly shaped bills and valleys, quadratic forms often fail to provide a good approximation. In such cases, the more sophisticated methods break down entirely or their performance seriously degrades.

Worse than degraded performance is the problem of local solutions. Almost all analytic methods, whether elementary or sophisticated, are easily trapped by local maxima: they generally fail to locate the globally best solution when there are’ many hills and valleys in the fitness surface. Least-squares, neural network predictive modeling gives rise to fitness surfaces that, although clearly analytic, are full of bumps, troughs, and other irregularities that lead standard analytic techniques (including back-propagation, a variant on steepest ascent) astray. Local maxima and other hazards that accompany such fitness surfaces can, however, be sidestepped by cleverly marrying a genetic algorithm with an analytic one. For titness surfaces amenable to analytic optimization, such a combined algorithm can provide the best of both worlds: fast, accurate solutions that are also likely to be globally optimal.

Some fitness surfaces are simply not amenable to analytic optimization. More specifically, analytic methods cannot be used when the fitness surface has flat areas or discontinuities in the region of parameter space where a solution is to be sought. Flat areas imply null gradients, hence the absence of a preferred direction in which to take a step. At points of discontinuity, the gradient is not defined; again, a stepping direction cannot be determined. Even if a method does not explicitly use gradient information, such information is employed implicitly by the optimization algorithm. Unfortunately, many fitness functions of interest to traders-including, for instance, all functions that involve net profit, drawdown, percentage of winning trades, risk-to-reward ratios, and other like items-have plateaus and discontinuities. They are, therefore, not tractable using analytic methods. Although the discussion has centered on the maximization of fitness, everything said applies as well to the minimization of cost. Any maximization technique can be used for minimization, and vice versa: Multiply a fitness function by - 1 to obtain an equivalent cost function; multiply a cost function by - 1 and a fitness function is the result. If a minimization algorithm takes your fancy, but a maximization is required, use this trick to avoid having to recode the optimization algorithm.