Preconditioning

Faking the epsilon

Burdened by a problem of oppressive size, any trick, honest or sly, is nice to know. I will tell you a trick that is widely used. Many studies are done ignoring (abandoning) the model styling regression (second fitting regression as follows):

$$\begin{array}{lll} \bold 0 &\approx & \bold F \bold A^{-1} \b... ...- \bold d \ \bold 0 &\approx & \epsilon \bold p \end{array}$$ (46)

Because we have a numerically poor idea of what epsilon should be, it is nice to be rid of it. The pragmatic person iterates the data-fitting regression only, watches the solution as a function of iteration, and stops when tired or (more hopefully) stops at the iteration that is subjectively most pleasing. The epsilon-faking trick does not really speed things. But, it eliminates the need for scan over epsilon. It also simplifies the coding (insert smiley emoticon).

Why does this crude approximation seem to work? The preconditioner is based on an analytic solution ( $\bold A^{-1}$ is an inverse) to the regularization, so naturally, early iterations tend to already fit the regularization, so early iterations are struggling instead to fit the data. The longer you run though, the better the data fit, and the more the actual regularization should be coming into play. But ongoing research often fails to run that far.

Figure 8 shows the idea that early iterations fit the straight lines. Straight lines honor the preconditioner. At later iterations the data fits better. Why do straight-line solutions honor the regularization? Refer to the discussion near Figure .

Preconditioning