Refactor Algorithm and Environment

[RemyDegenne]

Each round had two components: action and feedback. Now we have 3: observation -> action -> feedback.
If fits naturally settings like contextual bandits or prediction with expert advice, and it allows a nice refactor of IsBayesAlgEnvSeq, which can be written as

def IsBayesAlgEnvSeq (Q : Measure 𝓔) [IsProbabilityMeasure Q] (κ : Kernel (𝓔 × 𝓐) 𝓨)
    [IsMarkovKernel κ] (alg : Algorithm Unit 𝓐 𝓨)
    (E : Ω → 𝓔) (A : ℕ → Ω → 𝓐) (Y : ℕ → Ω → 𝓨)
    (P : Measure Ω) [IsProbabilityMeasure P] : Prop :=
  IsAlgEnvSeq (fun _ ↦ E) A Y (alg.comapObs (fun _ : 𝓔 ↦ ())) (bayesStationaryEnv Q κ) P

in which bayesStationaryEnv Q κ is a Environment 𝓔 𝓐 𝓨.

Depends on #226

[paulorauber]

In the spirit of getting Algorithm and Environment right once and for all, have you considered "bundling" the processes {O : ℕ → Ω → 𝓞} {A : ℕ → Ω → 𝓐} {Y : ℕ → Ω → 𝓨} into a structure or abbreviation? They almost always appear together, and conceptually this makes sense.

For example, if we had such an S : LearningProcess, then we could have S.IsAlgEnvProcess alg env P, and other random variables could be defined in the LearningProcess namespace. For example, we could have S.history n.

I am not saying this is definitely the way to go, but I want to hear your thoughts.

[RemyDegenne]

I think we should not do it. There would be projections S.action, S.obs everywhere in the code, which is heavier than just A or O. Also Mathlib uses a lot of function/process properties that would apply to projections but not to the structure unless we duplicate a log of API (Adapted, HasLaw, HasCondDistrib...), again forcing projections.

[paulorauber]

We would probably want to access the actions/observations/feedbacks through S.A/S.O/S.Y. I agree that the code becomes heavier when dealing with these variables, but it is also becomes lighter when dealing with derived variables, which are already much more verbose (for example, S.history n is an improvement over history O A Y n ). But I don't want to press the issue too much - I trust your judgement.

[paulorauber]

Thanks Rémy! I think comap is an elegant solution.

Regarding the Bayesian setting, attempting a further generalization to a prior over Environment (beyond just going from a prior over E to a prior over Kernel 𝓐 𝓨, which is what we talked about) will be quite revealing in the future.