Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions .github/workflows/llms_txt_snippet_sync.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
name: llms.txt Snippet Sync

# llms.txt is hand-maintained in the weaviate-io repo, so a snippet change here can
# strand a block in the published file. Only the weekly llms_txt_tests.yml job notices,
# which means the break surfaces days later. This gives the author the signal at PR time.
#
# It is advisory by design and never fails: when a snippet PR is opened, weaviate-io has
# not merged or deployed yet, so the live llms.txt legitimately cannot match. A blocking
# check would fire on every honest PR and would just be overridden.

permissions:
contents: read

on:
pull_request:
paths:
# Kept in step with SNIPPET_GLOBS in tests/test_llms_txt_code.py. Java and C#
# snippets live with their language suites, not under _includes/code/llms-txt/.
- "_includes/code/llms-txt/**"
- "_includes/code/java-v6/src/test/java/LlmsTxtTest.java"
- "_includes/code/csharp/LlmsTxtTest.cs"
- "tests/test_llms_txt_code.py"
- "tests/check_llms_txt_drift.py"

env:
PYTHON_VERSION: "3.11"

jobs:
check-snippet-sync:
name: Check llms.txt snippet sync
runs-on: ubuntu-latest
timeout-minutes: 5

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Fetch the PR base commit
# Shallow single-commit fetch: the check only needs the base tree to read the
# pre-change snippet files. If it fails the check reports a degraded warning
# rather than blocking.
continue-on-error: true
run: git fetch --no-tags --depth=1 origin ${{ github.event.pull_request.base.sha }}

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}

- name: Install pytest
# The check imports tests/test_llms_txt_code.py to reuse its matching logic, and
# that module imports pytest. Nothing else from the test suite is needed, so this
# stays far cheaper than the full setup-test-env composite.
run: python -m pip install --quiet "pytest>=8.3.5"

- name: Check llms.txt snippet sync
run: python tests/check_llms_txt_drift.py --base "${{ github.event.pull_request.base.sha }}"
21 changes: 19 additions & 2 deletions _includes/code/howto/search.filters.py
Original file line number Diff line number Diff line change
Expand Up @@ -538,9 +538,8 @@
filters=Filter.by_property("country").is_none(True) # Find objects where the `country` property is null
# highlight-end
)
print("despot. othing")
for o in response.objects:
print("despot"+o.properties) # Inspect returned objects
print(o.properties) # Inspect returned objects
# END FilterByPropertyNullState


Expand Down Expand Up @@ -573,6 +572,24 @@
},
)

# Test scaffolding below; not rendered in the docs.
# This block used to run against a local instance, where a write is queryable
# the moment `insert` returns. It now runs against a remote cluster, where it is
# not, so the example could query the collection before the object was visible
# and get back nothing. Wait (bounded) for the write to land, so the assertion
# tests the geo filter and not write visibility. The assertion stays exact.
import time
from weaviate.classes.query import Filter, GeoCoordinate

readiness_filter = Filter.by_property("headquartersGeoLocation").within_geo_range(
coordinate=GeoCoordinate(latitude=52.39, longitude=4.84),
distance=1000
)
for _ in range(30):
if len(publications.query.fetch_objects(filters=readiness_filter).objects) >= 1:
break
time.sleep(1)

# START FilterbyGeolocation
from weaviate.classes.query import Filter
from weaviate.classes.query import GeoCoordinate
Expand Down
5 changes: 5 additions & 0 deletions src/scripts/scarf.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ const scarfScript = {
src: "https://pixel.weaviate.cloud/a.png?x-pxid=a41b0758-a3a9-4874-a880-8b5d5a363d40",
referrerPolicy: "no-referrer-when-downgrade",
style: "display: none;",
// Decorative, hidden analytics beacon: empty alt plus aria-hidden keeps it
// out of the accessibility tree instead of leaving assistive technology to
// announce an image with no alt attribute.
alt: "",
"aria-hidden": "true",
},
};

Expand Down
14 changes: 8 additions & 6 deletions tests/README-INDEXABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Documentation content that is hidden behind JavaScript interactions (tabs, colla

### HTML structure tests (Part 1)

These tests fetch ~11 representative pages from all doc sections and check:
These tests fetch ~13 representative pages from all doc sections and check:

| Test | What it checks |
|------|---------------|
Expand Down Expand Up @@ -81,7 +81,7 @@ HTML structure tests always run. Agent tests only run if `ANTHROPIC_API_KEY` and

## Test pages

The suite tests 11 representative URLs covering all doc sections:
The suite tests 13 representative URLs covering all doc sections:

| Page | Features tested |
|------|----------------|
Expand All @@ -91,9 +91,11 @@ The suite tests 11 representative URLs covering all doc sections:
| `/weaviate/search/hybrid` | tabs, code |
| `/weaviate/connections/connect-cloud` | tabs, code |
| `/weaviate/config-refs/collections` | details, table |
| `/weaviate/concepts/data-import` | images |
| `/cloud/quickstart` | code, images |
| `/cloud/manage-clusters/create` | images |
| `/weaviate/concepts/data-import` | no structural features (200, meta tags, headings, LLM notice only) |
| `/cloud/quickstart` | code |
| `/cloud/manage-clusters/create` | no structural features (200, meta tags, headings, LLM notice only) |
| `/cloud/tools/query-tool` | images |
| `/weaviate/manage-collections/tenant-states` | images |
| `/query-agent/recipes/query-agent-ecommerce-assistant` | code |
| `/weaviate/search` | landing page |

Expand Down Expand Up @@ -129,4 +131,4 @@ TEST_PAGES = [
]
```

Available feature tags: `tabs`, `code`, `details`, `images`, `table`. Pages are parametrized — each feature tag enables the corresponding structural test for that page.
Available feature tags: `tabs`, `code`, `details`, `images`, `table`. Pages are parametrized — each feature tag enables the corresponding structural test for that page. Tag only what the page actually has: a page tagged `images` with no content image fails rather than passing quietly, which is what keeps the tags from going stale.
37 changes: 36 additions & 1 deletion tests/README-LLMS-TXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,8 @@ no Weaviate cluster.
2. Run that language's `test_llms_txt` and confirm it passes — **never** hand-write
a snippet without running it.
3. Copy the verified marked region **verbatim** into `weaviate-io/static/llms.txt`.
The PR-time sync warning (see *CI* below) flags this for you and prints the exact
block to paste.
4. New file? Add its path to the `test_llms_txt` parametrize list in the matching
`tests/test_*.py`.

Expand Down Expand Up @@ -182,12 +184,13 @@ after the matching `weaviate-io` change is live; otherwise mark with

## CI

Two separate workflows cover this directory:
Three separate workflows cover this directory:

| Workflow | What it runs |
|---|---|
| `.github/workflows/docs_tests.yml` | Per-language **execution** tests — `test_llms_txt*` in `test_python.py`, `test_typescript.py`, `test_java.py`, `test_csharp.py`. Rides the existing `pyv4` / `ts` / `java` / `csharp` / `agents` markers, so no separate job for these. |
| `.github/workflows/llms_txt_tests.yml` | The three **guard tests** in `test_llms_txt_code.py` — snippet coverage, version freshness, link validity. Single job `test-llms-txt`, runs `uv run pytest tests/test_llms_txt_code.py -m "llms_txt"`. |
| `.github/workflows/llms_txt_snippet_sync.yml` | The **PR-time warning** (`check_llms_txt_drift.py`). Advisory only, see below. |

The guard workflow triggers on:

Expand All @@ -204,6 +207,38 @@ Results post to Slack via the shared `./.github/actions/handle-test-results`
composite under `test-type: 'llms.txt'`, with `continue-on-error: true` on the
pytest step so the notification still fires when a guard fails.

### PR-time snippet sync warning

The guard workflow runs weekly, so a snippet change here can break the published
`llms.txt` days before anyone notices. `tests/check_llms_txt_drift.py` closes that
gap on the docs side. `.github/workflows/llms_txt_snippet_sync.yml` runs it on every
PR touching a file that matches `SNIPPET_GLOBS`, and it answers one question: once
this PR merges, which `llms.txt` code blocks would `test_llms_txt_snippets_are_covered`
no longer find? Those, and only those, are the blocks `weaviate-io/static/llms.txt`
has to update in lockstep.

It reuses this directory's matching logic (`SNIPPET_GLOBS`, the marker and fence
regexes, `_normalize`, `_load_llms_txt`), so it cannot disagree with the weekly job
about what "matches" means. Findings surface as GitHub warning annotations on the
changed snippet lines, plus a job summary carrying the new block to paste into
`llms.txt`.

**It is advisory and always exits 0.** When a snippet PR is opened, weaviate-io has
not merged or deployed yet, so the live `llms.txt` legitimately cannot match yet; a
blocking check would fire on every honest PR and would just get overridden. It also
stays quiet whenever there is nothing to do: weaviate-io shipped first and `llms.txt`
already carries the new block, only scaffolding outside the markers changed, or a
region was moved or renamed without its code changing.

Run it locally against uncommitted edits:

```bash
uv run python tests/check_llms_txt_drift.py --base HEAD
```

The reverse direction (an `llms.txt` edit in weaviate-io that no longer matches these
snippets) is checked by a workflow in the weaviate-io repo.

## Per-language gotchas

- **Python** — `text2vec_ollama` / `generative_ollama` take `api_endpoint` and
Expand Down
Loading
Loading