Repository navigation
Query layer (Q3a): the ask as a typed query over a declared answer, measured - #502
Open
SecureCloudGroup wants to merge 1 commit into
Open
SecureCloudGroup wants to merge 1 commit into
SecureCloudGroup wants to merge 1 commit into
Conversation
…l call, code-owned clauses, apply and interpretation, with the in-repo eval tool and fixtures
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step Q3a of the Round 20 plan. New code only; the flow does not use it yet (Q3b wires it in and removes the regex parsers one per measured delta).
What it is. The ask becomes one closed query object over the chosen declared answer:
{answer, time, where, order}from one local model call,{select, limit, agg, params}from code, every clause validated against the declared cell types, explicit time phrases decided by two recognizer libraries, and a code-rendered interpretation line ("Earthquakes · Magnitude > 4 · past 7 days · count") for the consent screen and the card footer.ni_query/recognize.py: Microsoft Recognizers-Text (community fork) + puckling (the Duckling port) merged into one closed span shape, behind a word-boundary filter and an entity mask; comparators from a documented adjacency table; no other ask regex.ni_query/prompt.py+plan.py: the measured prompt (literal skeleton, answer and parameter menus, three retrieved examples), the call throughni_forms.llm.chat_jsonwith a new optionalcheck=hook whose retry messages state the violated rule and never list a menu; coverage per clause; the rules floor when the model is unavailable.ni_query/normalize.py,clauses.py,apply.py,say.py: the v1 contract, the code-owned clauses, the row filter/cut/order/limit/count, the interpretation line.tools/ni-query-eval.py+tests/fixtures/ni_query/: the gold set (109 hand-labeled objects over the live cards of five blind sets), recorded model replies, 92 recorded samples, the DEV/TEST split and the few-shot pool;--repliesreplays deterministically,--execscores execution agreement,--gatewayruns the live model.Numbers. Deterministic replay of the recorded 9B replies: 50/60 fully right (few-shot TEST) and 54/82 (zero-shot ALL) under the v1 contract, matching the lead's independent re-score to the row; 53/60 and 65/82 with the rules floor; execution agreement 46/59 and 55/80. Live through the local gateway with byte-identical prompts: 51/60, 53/60 with the floor; valid 68 first try / 9 after the rule-only retry / 2 never. Per clause live: answer 55, select 57, where 52, time 51, order 55, limit 58, agg 60 of 60.
Dependencies.
ms-recognizers-text-suite==1.0.1(MIT, six sub-packages) andpuckling==0.5.0(Apache-2.0) with their transitive pins (emoji, grapheme, multipledispatch, datedelta) inrequirements.lockandTHIRD_PARTY_LICENSES.md; both imported lazily.Gates (Docker
smartbrain_3000:dev): ruff clean; targeted 183 passed; replay parity 109/109 and 79/79; full suite 4,280 passed, 18 skipped on the identical scratch tree.Open for Q3b:
answerreplacesselect_answers;timereplaces_window_from_textbound to the window ops with the answer's direction and the clock parameters;wherereplaces the subject row filter family with aparam_boundspush-down (Library key);aggreplaces the count rule;spec.interpretation; a messages adapter for the flow'scall_model. One ruling needed: "over the last year" as calendar 2025 (the recognizers) or rolling 365 days (the gold labels).