[kmr-rohit]

Why

Two questions from a maintainer got "could not find" answers:

• "do we have a helm chart?" The agent answered without searching, and the answer is not indexed: it lives in kubeflow/community-distribution (README and Helm chart READMEs).
• "Are the new workspaces in the new release?" The Workspaces pages are indexed, but "latest" queries were filtered to release notes only.

Looking at the cluster showed the wider problem: nothing ingests on a schedule (docs last indexed 30 Aug, issues and code 17 Aug), and the image the docs pipeline runs in (ghcr.io/kubeflow/docs-rag-ingest) was never published.

What changes

Ingest

• The docs, issues and code pipelines become Kubeflow Pipelines recurring runs, registered by an Argo CD PostSync Job (ingest_schedule.py, chart values ingest.*, on in production only).

  Schedule	When (UTC)	Source
  docs-website	daily 02:00	kubeflow/website docs
  docs-distribution	daily 02:30	kubeflow/community-distribution Markdown, doc_type: distribution, same collection
  issues	every 6 h	issues updated in the last 24 h, ten repositories
  code	Sunday 03:00	community-distribution manifests and Helm charts, without CRDs

• CI builds and pushes the ingest image per commit; promote.sh checks it exists.

• The docs pipeline fails on an incomplete download and removes pages deleted upstream. It refuses to remove more than 20% of a source.

• Badge-only chunks are dropped (they were the top hits for the Helm question).

Retrieval

• "Latest / new release" questions return release notes and general hits.
• With a reranker configured, 20 candidates are fetched for it instead of top_k.
• The docs prompt requires a search for product questions that do not name Kubeflow.
• Both questions are added to the eval dataset.

Platform fixes found on the way

• reranker Argo CD app: RERANKER_URL pointed at a service that was never deployed.
• Langfuse retention: DELETE ... now() is rejected on replicated ClickHouse tables; the job has failed weekly since.
• Gateway: httpsRedirect on port 80 blocks cert-manager's HTTP-01 challenge, so kagent-ui-cert has not renewed since 28 Oct and expires 27 Nov. Port 80 now serves only the challenge; the routes match port 443.

Tested

• pytest: 190 passed (30 new in tests/test_rag_quality_ingest.py); ruff clean; all three pipelines compile; Helm lint and render with ingest on and off, TLS on and off.
• Downloaded and chunked the real kubeflow/community-distribution: 96 files, 1006 chunks.
• Server-side dry-run of the new manifests against the demo cluster.

Not tested yet

• A pipeline run with the new ingest image, and registering the recurring runs against the live Pipelines API.
• Reranker latency on CPU.
• Certificate renewal and the retention delete (both need the change synced).
• /eval with the prompt change.

After merge

1. The docs-rag-ingest package on GHCR must be public (or pullable from the kubeflow namespace).
2. After promotion: run the issues backfill once (--set updated_within_hours=0) and delete code_rag rows with repo_name == "kubeflow/manifests".
3. The Argo CD and Langfuse gateways in deploy/envs/prod/ingress/ have the same redirect; follow-up once this fix is confirmed on the chatbot host.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chasecadet for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tarekabouzeid]

/eval

[github-actions]

Eval of 56e498ca requested by @tarekabouzeid. Argo CD deploys it to the preview within a few minutes; the comparison with main is posted here when the run finishes (usually 10–20 minutes).