Hierarchies in the documentation drawn by the 'tree' directive

[pytooling-claude]

Documentation

• Directory structures are drawn by pyTooling.Sphinx' tree directive instead of indented text in a code block:
  📂 directory, 📦 Python package, 🐍 Python file, 📄 other file. Entries can be folded.
  • RepositoryStructure.rst, index.rst (example pipeline), UnitTesting.rst, and the example directory structures of
    Parameters.rst (2) and CompletePipeline.rst (4).
  • index.rst named the documentation directory docs/; it is doc/, as everywhere else.
• CompletePipeline.rst: new topic Workflow Hierarchy - the 23 jobs of CompletePipeline.yml in job order, each with
  the reusable workflow it calls (linked to its job template page, the job name as description) and the actions that
  workflow uses; composite actions (🧩) are followed into their own steps, third-party actions (🔗) and the MiKTeX
  container image (🐳) are leaves. Initially only the job level is expanded. The tree (106 entries) is generated from
  the YAML files by the script below; after a workflow changes, run it and replace the tree.
• doc/conf.py: package option maxlistdepth=10 for sphinx. A tree is rendered as nested bullet lists in LaTeX, and
  the namespace package's directory tree has 5 levels - more than LaTeX' default of 4 ("Too deeply nested").
• Not converted: the linear job chain in Development.rst (not a tree), log output, a Markdown default value, and the
  Dependencies dropdown on the CompletePipeline page (a nested list with pip/apt/MSYS2 entries - but outdated, see
  Known Issues).
• Local build: no new HTML or LaTeX warnings; every tree has as many entries as its source; all 23 job links resolve.

Known Issues

• The Dependencies dropdown on the CompletePipeline page is outdated against the YAML files: it omits
  CheckReleaseVersion.yml, lists nothing for Parameters.yml (which uses checkout and setup-python), and omits
  ComputeRequirements/ComputePacboyPackages under UnitTesting, StaticTypeCheck and ApplicationTesting.
• In the PDF, trees have no icons, so the icon legend above the workflow tree doesn't apply there.

Generator script (WorkflowTree.py)

"""
Print the hierarchy of reusable workflows and actions a pyTooling/Actions workflow uses, as a ReST 'tree' directive.

Usage: python3 WorkflowTree.py <Actions checkout> [<workflow file name> ...]   (default: CompletePipeline.yml)

* A job calling a reusable workflow of pyTooling/Actions becomes a node: the file, linking to its job template page;
  its description is the calling job's name.
* A step's 'uses:' becomes a child; actions of pyTooling/Actions (.github/actions/*) and pyTooling's artifact actions
  are followed into their 'action.yml'. Third-party actions are leaves.
* A job's 'container:' becomes a leaf too.
* Entries are listed in job/step order, each action once per file.
"""
from pathlib import Path
from re      import compile as re_compile
from subprocess import run
from sys     import argv

from yaml import safe_load

ROOT = Path(argv[1]) if len(argv) > 1 else Path(".")
WORKFLOWS = argv[2:] or ["CompletePipeline.yml"]

# pyTooling's composite actions living in other repositories -> local clone, branch = the ref used
CLONES = {
	"pyTooling/upload-artifact":   Path("/work/upload-artifact"),
	"pyTooling/download-artifact": Path("/work/download-artifact"),
}

OWN_WORKFLOW = re_compile(r"^(?:pyTooling/Actions/)?\.?/?\.github/workflows/(?P<file>[^@]+?)(?:@(?P<ref>.+))?$")
OWN_ACTION =   re_compile(r"^(?:pyTooling/Actions/)?\.?/?\.github/actions/(?P<name>[^@]+?)(?:@(?P<ref>.+))?$")

INPUT = re_compile(r"^\$\{\{\s*inputs\.(?P<input>\w+)\s*\}\}$")

# job template page labels, found in doc/JobTemplate/**
LABELS = set()
for rst in (ROOT / "doc").rglob("*.rst"):
	for line in rst.read_text(encoding="utf-8").splitlines():
		if line.startswith(".. _") and line.endswith(":"):
			LABELS.add(line[4:-1])


def workflowEntry(file: str) -> str:
	label = f"JOBTMPL/{Path(file).stem}"
	return f":ref:`{file} <{label}>`" if label in LABELS else f":file:`{file}`"


def actionName(data: dict) -> str:
	"""The action's 'name:' without a leading emoji."""
	name = data.get("name", "")
	return name.lstrip("".join(c for c in name if not (c.isalnum() or c in "'\"("))).strip() if name else ""


def stepUses(steps: list) -> list[str]:
	result = []
	for step in steps or []:
		if "uses" in step and step["uses"] not in result:
			result.append(step["uses"])
	return result


def actionChildren(uses: str, indent: int, lines: list[str]) -> None:
	pad = "  " * indent
	if (match := OWN_ACTION.match(uses)) is not None:
		name = match["name"]
		data = safe_load((ROOT / ".github/actions" / name / "action.yml").read_text(encoding="utf-8"))
		lines.append(f"{pad}+ :ghsrc:`{name} <.github/actions/{name}>` | {actionName(data)}")
		for child in stepUses(data.get("runs", {}).get("steps")):
			actionChildren(child, indent + 1, lines)
		return

	repo, _, ref = uses.partition("@")
	if repo in CLONES:
		text = run(["git", "-C", str(CLONES[repo]), "show", f"origin/{ref}:action.yml"], capture_output=True, text=True,
		           check=True).stdout
		data = safe_load(text)
		lines.append(f"{pad}+ :gh:`{uses} <{repo}>`")
		for child in stepUses(data.get("runs", {}).get("steps")):
			actionChildren(child, indent + 1, lines)
		return

	lines.append(f"{pad}> :gh:`{uses} <{repo}>`")


def workflowChildren(file: str, indent: int, lines: list[str]) -> None:
	data = safe_load((ROOT / ".github/workflows" / file).read_text(encoding="utf-8"))
	seen = []
	for jobName, job in data["jobs"].items():
		if "uses" in job:
			match = OWN_WORKFLOW.match(job["uses"])
			child = match["file"]
			pad = "  " * indent
			lines.append(f"{pad}- {workflowEntry(child)} | job ``{jobName}``")
			workflowChildren(child, indent + 1, lines)
			continue

		if "container" in job:
			image = job["container"] if isinstance(job["container"], str) else job["container"]["image"]
			if (match := INPUT.match(image)) is not None:
				image = data["on"]["workflow_call"]["inputs"][match["input"]]["default"] if "on" in data else \
				        data[True]["workflow_call"]["inputs"][match["input"]]["default"]
			repo = image.partition(":")[0]
			if image not in seen:
				seen.append(image)
				lines.append(f"{'  ' * indent}= :dockerhub:`{image} <{repo}>` | container image")
		for uses in stepUses(job.get("steps")):
			if uses not in seen:
				seen.append(uses)
				actionChildren(uses, indent, lines)


for workflow in WORKFLOWS:
	print(".. tree::")
	print("   :root-icon:       📄")
	print("   :node-icon:       📄")
	print("   :leaf-icon:       📄")
	print("   :icons:           + 🧩, > 🔗, = 🐳")
	print("   :expanded-levels: 1")
	print()
	lines = [f"- :file:`{workflow}`"]
	workflowChildren(workflow, 1, lines)

	# align the descriptions of the root's children (the jobs) in one column
	jobs = [i for i, line in enumerate(lines) if line.startswith("  - ") and " | " in line]
	width = max(len(lines[i].partition(" | ")[0]) for i in jobs)
	for i in jobs:
		text, _, description = lines[i].partition(" | ")
		lines[i] = f"{text:<{width}} | {description}"
	print("\n".join(f"   {line}" for line in lines))
	print()

Run: python3 WorkflowTree.py <Actions checkout> [CompletePipeline.yml ...], indent the output by 3 spaces inside the
topic. It follows pyTooling/upload-artifact and pyTooling/download-artifact into their action.yml from local
clones next to the checkout.

---

Related Issues and Pull-Requests

• Stacked on Documentation rendered with pyTooling.Sphinx instead of sphinx-reports #297 - merge that one first.
• Code coverage: the data model's hierarchies as a tree edaa-org/pyEDAA.Reports#279 - the first conversion to tree.
• tree: descriptions in a second column pyTooling.Sphinx#29 - descriptions in a second column.

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 complexity · 0 duplication

Metric	Results
Complexity	0
Duplication	0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}