Each analysis project should be fully self-contained

[lwjohnst86]

My suggestion would be that each project is self-contained to taking the raw data and outputting the final product. Don't make an intermediary dataset that other products depend on, since that dependency will become an issue as time goes on.

Originally posted by @lwjohnst86 in dp-next/wp1-ros#100 (comment)

[lwjohnst86]

This makes a lot of sense, and I think everyone working on main is the right starting point, at least. On one hand, I agree that self-contained projects are definitely easier to keep clean, but otoh if we end up with multiple projects on the same data that REQUIRE the cleaned datasets to be identical (e.g., for comparability reasons) then having each project contain copies of the exact same data-cleaning pipeline can become equally (or more) messy to maintain. This could be solved with git submodules, but that adds complexity of course and requires effective communication around updating dependents whenever the submodule changes. --- @zheer-kejlberg in dp-next/wp1-ros#100

[lwjohnst86]

Just moving this discussion here @zheer-kejlberg. I think that if there is so much dependency on a specific cleaned dataset, then there should be a specific project/"product" that builds that output. But the problem comes when an analysis in one paper needs something different from previous analysis paper, how do you keep track of which version each paper is using for the "cleaned data product". It quickly gets very complicated. Hence why I suggest keeping it fully self-contained, even if there is some code duplication across projects. At least that duplication can more easily allow modifications and customisations without impacting others work.

[zheer-kejlberg]

Good points, fully agree. A bit unclear what can be considered an intermediary dataset to be avoided, then, but if it's mainly a question of such datasets being confined to their own projects, then I think that's a good solution.