Skip to content

Repository cleanup and package consolidation - #109

Merged
glrs merged 33 commits into
mainfrom
dev
Oct 12, 2026
Merged

glrs merged 33 commits into
mainfrom
dev

Conversation

@glrs

@glrs glrs commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator

This PR brings the repository-cleanup and package-consolidation work from dev into main, preparing Yggdrasil for v0.5.0.

The central change is the retirement of the top-level lib package. All maintained code now lives under yggdrasil, with responsibilities reorganized into explicit packages. This makes shared tools and contracts directly importable by realm packages and consumers such as Bifrost, without relying on legacy import aliases or checkout-specific path manipulation.

The current planning and execution model is preserved. Import paths and legacy integration mechanisms intentionally change.

Repository Cleanup

  • Removed unused internal realms, obsolete handlers and registries, legacy reporting code, and superseded entry scripts.
  • Removed obsolete configuration files, tests tied to retired implementations, and unused runtime dependencies.
  • Retained deprecated components whose removal requires separate architectural work. This is not a wholesale redesign of storage, execution, or daemon lifecycle.

Package Consolidation

  • Moved maintained lib code into yggdrasil, organized by responsibility across configuration, Flow, watchers, CouchDB access, storage, operations, daemon, and CLI packages.
  • Removed the legacy import-alias mechanism. There is no compatibility shim for lib.*.
  • Replaced YggdrasilUtilities with module-level workspace helpers and consolidated external-system defaults.
  • Moved YggdrasilEvent into watchers/events.py and CouchDBCheckpointStore into storage/couchdb/checkpoint_store.py.
  • Made realm-facing imports lightweight: importing yggdrasil.flow or yggdrasil.watchers does not load the CouchDB client or configure process-wide logging.
  • Moved logging level abbreviations and third-party logger configuration into explicit configure_logging() setup.
  • Retired ygg.handler discovery in favor of ygg.realm registration through RealmDescriptor.

Packaging And Verification

  • Replaced the explicit package list with bounded setuptools package discovery, using regular packages rather than implicit namespace packages.
  • Preserved bundled assets, py.typed, and Git-tag-based versioning.
  • Added regression coverage for package structure, lightweight imports, watcher exports, logging setup, and shared defaults.
  • Validated wheel construction from an sdist without Git metadata, installed-package imports, realm entry-point registration, and daemon startup/shutdown.
  • Excluded generated build/ output from mypy checks.

Documentation And CI

  • Updated architecture, configuration, realm-authoring, and developer documentation to match the implemented package layout and API boundaries.
  • Promoted the design records and documented implementation outcomes and departures.
  • Replaced unnecessary references to the unpublished technical-debt ledger with self-contained explanations.
  • Updated GitHub Actions dependencies and excluded a deleted placeholder-credential fixture from historical secret scanning.

Upgrade Considerations

  • Consumers must replace lib.* imports; legacy aliases are no longer available.
  • Realms must register through ygg.realm; ygg.handler entry points are not discovered.
  • Logger-name filters must account for the new yggdrasil.* module paths.
  • Development-database test-realm plans referencing lib.realms.test_realm must be deleted and regenerated.
  • Development checkouts require an installation, normally pip install -e ., rather than automatic repository-path insertion.
  • Legacy ygg-mule.py and ygg_trunk.py entry scripts are removed; use the supported yggdrasil CLI.

glrs added 30 commits October 7, 2026 13:37
Remove lib/realms/smartseq3/, lib/realms/tenx/ and lib/realms/delivery/,
which predate the realm descriptor / plan / Engine model and are not
imported by the core or its tests. This includes the 13 tracked .DS_Store
files under smartseq3/report/assets/.

lib/realms/ now holds only the dev-only test_realm; production realms are
external packages. README.md, CLAUDE.md and docs/architecture/overview.md
are updated to say so.

The removed code can be retrieved from this commit's parent:
  git checkout <this-commit>^ -- lib/realms/<name>
Remove code that served only the legacy realms, or nothing at all:
- module_registry.json, lib/core_utils/module_resolver.py and its tests
- ygg_trunk.py and ygg-mule.py, superseded by the yggdrasil CLI
- the root config.json, superseded by main.json
- lib/base/ (AbstractProject, AbstractSample)
- lib/handlers/ (BestPracticeAnalysisHandler and the base_handler shim)
- lib/realms/test_realm/watcher.py (ScenarioDocWatcher, never wired)
- lib/module_utils/ngi_report_generator.py and its tests

ProjectDBManager no longer loads the module registry. fetch_changes(),
which only mapped library-construction methods to realm modules, is
removed with its tests. YggdrasilCore and test_base_handler import
BaseHandler from yggdrasil.flow.base_handler, and the commented-out
registry-based run_once block in the core is deleted.

Runtime dependencies go from sixteen to seven: appdirs, cattrs,
ibmcloudant, panzi_json_logic, requests, rich and watchdog. cattrs is
now declared (pinned to 26.2.1) because yggdrasil/flow/utils/codec.py
uses it when it is installed. requirements/lock.txt is regenerated.

External code that imports yggdrasil.handlers.base_handler (resolved
through the lib alias) must switch to yggdrasil.flow.base_handler.
…unctions; move normalize_url to config.external_systems
…functionality (note OpsWriter's own endpoint resolution)
…o enhance functionality and improve performance
yggdrasil/flow/utils was the only directory with Python modules and no __init__.py, and namespaces = true was set to pick it up. It now has a docstring-only initializer, and packages.find runs with namespaces = false
Six docstrings cited entries of docs/TECH_DEBT_LEDGER.md, a local file the repository does not track. Each now states the limitation itself.

The watchers package docstring describes AbstractWatcher, PlanWatcher and SeqDataWatcher instead of calling them deprecated after a certain plan phase.
@glrs
glrs merged commit 000d596 into main Oct 12, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant