For difficult or ill-conditioned optimization problems, the line search can sometimes return a zero step size α = 0.0. In some cases, this causes the optimization to get stuck in a loop where it maxes out the line search iterations at every optimization step without ever moving, until it hits the optimization maxiter.
At the very least, a step size of exactly zero should probably get a dedicated warning to alert the user that something is going quite wrong. Moreover, sometimes this can be "solved" by restarting the optimization from exactly the same point where it was failing, indicating that forcing a restart (for ConjugateGradient) or clearing/reducing the memory (for LBFGS) can be helpful when optimizations get stuck. Finally, having to ability to allow a forced step even though the line search didn't converge properly could help kick the optimization out of a problematic region.
I'm not sure what the proper approach to this issue would be, but I think all of the suggestions could at least be better than the current default behavior.
For difficult or ill-conditioned optimization problems, the line search can sometimes return a zero step size α = 0.0. In some cases, this causes the optimization to get stuck in a loop where it maxes out the line search iterations at every optimization step without ever moving, until it hits the optimization
maxiter.At the very least, a step size of exactly zero should probably get a dedicated warning to alert the user that something is going quite wrong. Moreover, sometimes this can be "solved" by restarting the optimization from exactly the same point where it was failing, indicating that forcing a restart (for
ConjugateGradient) or clearing/reducing the memory (forLBFGS) can be helpful when optimizations get stuck. Finally, having to ability to allow a forced step even though the line search didn't converge properly could help kick the optimization out of a problematic region.I'm not sure what the proper approach to this issue would be, but I think all of the suggestions could at least be better than the current default behavior.