fix(tests): exempt hidden analytics beacons from the alt-text check - #488
Draft
g-despot wants to merge 6 commits into
Draft
fix(tests): exempt hidden analytics beacons from the alt-text check#488g-despot wants to merge 6 commits into
g-despot wants to merge 6 commits into
Conversation
The indexability suite already exempted the analytics beacon by hostname
("static.scarf.sh"). That predicate went stale when the pixel moved to our own
CNAME (pixel.weaviate.cloud), so test_images_have_alt_text started failing on
/weaviate/concepts/data-import, /cloud/quickstart and
/cloud/manage-clusters/create.
Repair the exemption structurally instead of by hostname: skip images hidden
from both rendering and the accessibility tree (inline display:none or
visibility:hidden, aria-hidden="true") and 1x1 tracking pixels. Alt text is
meaningless on an image no reader, screen reader or crawler can reach, and the
rule survives the next domain move. The scarf.sh case is kept. Content images
missing alt text are still caught.
Also mark the beacon itself as decorative in scarf.js with alt="" and
aria-hidden="true", which is the correct markup for a hidden pixel.
![orca-security-eu[bot]](https://avatars.githubusercontent.com/in/276244?s=60&v=4){kind=small}
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
 Passed |
Infrastructure as Code |  0  0  0  0 |
View in Orca |
| Passed |
SAST | 0 0 0 0 |
View in Orca |
| Passed |
Secrets | 0 0 0 0 |
View in Orca |
| Passed |
Vulnerabilities | 0 0 0 0 |
View in Orca |
The FilterByGeolocation block deletes a collection, recreates it, inserts one object and immediately queries it with a geo filter. That was safe while the block ran against a local instance, but it now runs against the shared remote cluster, where the object is not always visible the moment insert returns. The query then matched nothing and `assert len(response.objects) == 1` failed. The expected count of 1 is correct: the object sits 403 m from the query centre and the radius is 1000 m. Verified against a local instance (match cut-off falls between 400 m and 405 m) and against the CI run of 2026-07-18, which passed and printed exactly that one object. Add a bounded readiness wait to the test scaffolding, above the START marker so the rendered example is unchanged, and keep the assertion exact.
test_images_have_alt_text collected every img in the document, so on pages with no content images the only survivor was the Ask AI button logo in the navbar. The check passed on three pages by asserting on site furniture that is identical everywhere, and `len(content_images) > 0` held at exactly 1, so any navbar change would have flipped it red for no reason connected to those pages. Collect images from the main content region instead. That drops the navbar and footer without calling the Ask AI logo decorative, and it leaves the existing decorative filter untouched. Two of the labelled pages then have no content image at all and the third has only language-tab icons, so the check now skips them by name with a reason and records them in PAGES_MISSING_CONTENT_IMAGES. The label stays on the pages. Any other page labelled `images` without one fails, so a stale label cannot pass quietly. Add two pages that do carry screenshots, so the check enforces alt text on real images: 3 on the query tool page and 4 on tenant states. Verified that stripping their alt text fails the check, and that stripping alt from the Ask AI logo does not.
The Python example for filtering on a null property carried two debug prints:
`print("despot. othing")` and `print("despot"+o.properties)`. Both sit inside
the START/END markers, so both are published on the filters page today.
The second one is not just noise, it cannot run: concatenating a str and a dict
raises TypeError. It only stays quiet in CI because the query returns no objects
on the test cluster, so the loop body never executes.
Restore the line to the plain property print used by every sibling example.
llms.txt is hand-maintained in weaviate-io, but its code blocks must match the START/END regions of the snippets in this repo. The only enforcement was the weekly test_llms_txt_snippets_are_covered job, which fetches the live file. So PR #485/#487 moved the quickstart snippets to `data.ingest`, nothing here told the author that weaviate-io/static/llms.txt had to move too, and the break surfaced days later in a scheduled run. It is still red: 2 of 51 blocks. Add tests/check_llms_txt_drift.py, run by llms_txt_snippet_sync.yml on any PR touching a file that matches SNIPPET_GLOBS. It answers one question: once this PR merges, which llms.txt blocks would the weekly coverage test no longer find? It reuses that test's globs, regexes, _normalize and _load_llms_txt, so the two cannot disagree about what "matches" means. The check is advisory and always exits 0. When a snippet PR is opened, weaviate-io has not merged or deployed yet, so the live llms.txt legitimately cannot match; a blocking check would fail every honest PR and would be routinely overridden, which teaches people to ignore it. Findings are GitHub warning annotations on the changed marker lines plus a job summary carrying the exact block to paste into llms.txt. It stays quiet when there is nothing to do: weaviate-io shipped first and llms.txt already carries the new block, only scaffolding outside the markers changed, or a region moved or was renamed with its code intact. It degrades to a generic reminder, never a failure, if the base ref or llms.txt cannot be read. The path filter also covers the Java and C# snippets, which live with their language suites rather than under _includes/code/llms-txt/. llms_txt_tests.yml is untouched; the scheduled job behaves exactly as before.
The pages were labelled `images` on the assumption that they should carry screenshots and were missing them. They were never meant to have any, so there is no coverage to preserve and the label is simply wrong. Removing it also removes the reason for PAGES_MISSING_CONTENT_IMAGES, a second hand-maintained list that contradicted the feature labels a few lines above it. The hard failure for a page labelled `images` with no content image stays, and is now the only enforcement: a stale label fails instead of being skipped by name. Two of the three pages have no structural features left. Keep them in TEST_PAGES, because four checks (200, meta tags, heading hierarchy, LLM notice) parametrize over every entry regardless of feature, and dropping the entries would lose that coverage. An empty feature set would have exempted them from the h2 assertion, which keyed "content page" off a non-empty feature set, so record landing pages explicitly in LANDING_PAGES instead. That leaves _features_for without a caller, so it goes too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes 3
test_images_have_alt_textfailures on main: /weaviate/concepts/data-import, /cloud/quickstart, /cloud/manage-clusters/create.The suite already exempted analytics beacons, but by hostname (
static.scarf.sh). That check went stale when the beacon moved to our own CNAME (pixel.weaviate.cloud), so it began flagging a hidden 1x1 tracking pixel as a content image missing alt text.Replaces the hostname match with a structural check (display:none, visibility:hidden, aria-hidden, or 1px dimensions) so it will not go stale on the next domain change. Also sets
alt=""andaria-hiddenon the injected beacon in scarf.js, which is the WCAG-correct markup for a decorative hidden pixel.Local: the 3 target tests pass, full
-m indexabilityrun is 63 passed.Note: the scarf.js half only takes effect after deploy, since the suite reads the live site.
Draft pending the verifier gate.