You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add individual learning rate parameters to AdamOptimizer #84
Because many parameters have wildly differing "natural" scales, it makes sense to make it possible to adjust the learning rates on a per-parameter basis.
The current way this is handled is by introducing scaling factors for key parameters where this is a problem - e.g. the probe and translation offsets - to ensure that all parameters have a "typical value" around 1.
I think we can do better! I think the best way to handle this would be to have a "learning rate factor" set per-parameter, which defaults to a sensible value. Doing this in a general way could make it easier to tune reconstructions to converge faster, as I highly suspect a major cause of slow convergence at the moment is a mismatch in typical step size.
I know that at least one member of the CXI group has done work in this direction already, I think any discussion about a sensible way to introduce this while minimizing backward compatibility issues and keeping the simple "you only need to think about very few parameters if you don't want to" philosophy in place.
Activity
[-]Add individual learning rate parameters to Adam[/-][+]Add individual learning rate parameters to AdamOptimizer[/+]allevitan commentedon Aug 4, 2026
Closed by #91