Optimizer precision

[benikm91]

No description provided.

[benikm91]

@marcelluethi As discussed today, I will add tests before we merge.

[marcelluethi]

Thanks for implementing this. This looks like a very good solution to me.

[marcelluethi]

Will this not be confusing on single tensors, as they are both tensor trees and tensors and consequently will have +, +! and ++!. Another problem might be that ! usually indicates broadcasting and has here a different meaning.

[benikm91]

This mostly mirrors what we have in ValueOps for Tensors. While I think we can improve this overall, I propose moving it to a future PR and merging it as is.

Currently:
+: Add two tensors of the same shape
+!: Add two tensors, broadcast lower-rank one
++: Add two tensor trees of the same structure
++!: Add Tensor0/Double to a tensor tree by broadcasting it for each structure

So, in my mind, ++! does broadcasting (just limited to Tensor0, as it is the only structure broadcastable to all Tensor shapes in the tree). But you are probably right that it's suboptimal; I think, as is, we can write "t0 ++! 5" but not "t0 +! 5" with t0 being Tensor0... Also, ValueOps provides Float extension methods, and FloatTree provides Double extension methods... Let's address this in a future clean-up.

[marcelluethi]

Thanks for the fix. I think it is now the right way to handle the precision of the optimizers. I also like the optimizer parameters are just Double and hence easy to fix. The only thing I am not sure about are the !++ extension methods. However, I am fine with merging if this is not an accident but a conscious decision.