Skip to content

feat(schema): make the redis graph scan skippable, and fix skip_redis - #35

Open
YaphetKG wants to merge 2 commits into
developfrom
skip-redis-schema-scan
Open

YaphetKG wants to merge 2 commits into
developfrom
skip-redis-schema-scan

Conversation

@YaphetKG

@YaphetKG YaphetKG commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Problem

Building the schema scans the redis graph for node types and a summary. Measured against a 76M-edge FalkorDB graph:

measured
full schema build 833 s
of which the summary query MATCH (c) RETURN DISTINCT labels(c) as types, count(c) as count 99 s
node types resolved 41

Three multipliers on top of that:

  • SchemaFactory._cached is a class attribute, so the build is per worker process.
  • The refresh thread repeats it every update_interval, hardcoded to 20*60. An 833 s build on a 1200 s sleep means each worker spends ~40% of its life rebuilding the schema, indefinitely.
  • A worker's first request pays it twice concurrentlySchemaFactory.__init__ builds synchronously, while the update_cache_loop thread it just started immediately builds again.

Observed effect: raising gunicorn from 1 to 16 workers made a client crawl make zero progress for 30+ minutes, with the tranql pod pinned at its 2-core limit and redis idle at 7m.

None of that scan is used by redis: queries

SelectStatement.execute reads only:

all_schemas = interpreter.schema.config['schema']
redis_key = [k for k in all_schemas if 'redis' in all_schemas[k] and self.service.startswith(k + ':')]
redis_connection_details = all_schemas[redis_key]['redis_connection_params']

That's schema.config, loaded from schema.yaml. It then hands the query graph to PLATER's GraphInterface.answer_trapi_question. The scanned type map lives in schema.schema, which this path never reads — generate_questions uses the parsed query's concept.type_name/concept.curies, and plan()/execute_plan() is the /schema branch.

skip_redis already existed, but was unreachable and broken

  • Unreachable: TranQL.__init__ constructed SchemaFactory(...) without it, so it was always the default False.
  • Broken: the scan is also what sets metadata['schema'] (line 309), and schema_data = metadata['schema'] is read unconditionally later in the same loop (line 322). So skip_redis=True raised KeyError('schema') for any redis-backed entry — it can only ever have run against configs with no redis schema. My test caught this; skipping now leaves an empty type map rather than an absent key.

Change

  • Wire SKIP_REDIS_SCHEMA through TranQL.__init__SchemaFactorySchema, and make the skip path coherent.
  • Expose SCHEMA_UPDATE_INTERVAL, because 20 minutes is only safe relative to how long the scan takes.
  • Add util.as_bool. Config.__getitem__ returns environment overrides verbatim, so SKIP_REDIS_SCHEMA=false arrives as the string "false" — and bool("false") is True. Without coercion the off switch would mean on. It raises on anything uninterpretable rather than silently picking a branch.

Default stays false, so current behaviour is preserved. Deployments whose queries all name a redis: source can turn it on.

Caveat

TranQL's own /search endpoint infers its default indexes from the scanned schema (api.py: schema.schema[redis_schema_name]["schema"].keys()). Enabling SKIP_REDIS_SCHEMA leaves that empty, so it should stay off for any deployment using /search. The query endpoint is unaffected.

The other flags read via Config.get (ASYNCHRONOUS_REQUESTS, NAME_BASED_MERGING, …) have the same latent string-boolean issue if set from the environment. Not touched here to keep the diff reviewable.

Tests

tests/test_skip_redis_schema.py, 22 tests, no redis or graph needed:

  • as_bool across "false"/"False"/"0"/"no"/"off"/"" and the truthy forms, real bools, None + default, and that garbage raises
  • skip_redis=True does not construct RedisAdapter, while config['schema'][...]['redis_connection_params'] still loads — that's what the query path actually reads
  • skip_redis=False still constructs it, so this fix cannot silently disable the scan for everyone

tests/test_tranql.py is 8 failed / 41 passed both before and after, verified by running the same suite against a pristine origin/develop tree — the same eight test names fail either way, so they are pre-existing.

Building the schema scans the redis graph for its node types and a summary.
Measured against a 76M edge FalkorDB graph that scan takes 833s -- the
`MATCH (c) RETURN DISTINCT labels(c) as types, count(c) as count` summary
alone is 99s -- and it is paid per worker process, because
SchemaFactory._cached is a class attribute. The refresh thread then repeats
it every update_interval, so with the hardcoded 20 minutes each worker spent
roughly 40% of its life rebuilding, indefinitely. A worker's first request
pays it twice concurrently: __init__ builds synchronously while the thread
it just started immediately builds again.

None of that work is read by a query naming a `redis:<graph>` source.
SelectStatement.execute reads only
schema.config['schema'][key]['redis_connection_params'], which comes from
schema.yaml, then hands the query graph to PLATER's GraphInterface. The
scanned type map lives in schema.schema, which that path never touches.

A skip_redis flag already existed but was unreachable and broken:

- Unreachable: TranQL.__init__ constructed SchemaFactory without it, so it
  was always the default False.
- Broken: the scan is also what sets metadata['schema'], and that key is
  read unconditionally later in the same loop, so skip_redis=True raised
  KeyError('schema') for any redis backed entry. It presumably only ever
  ran against configs with no redis schema. Skipping now leaves an empty
  type map instead of an absent key.

So this wires SKIP_REDIS_SCHEMA through and makes it actually work.
Default stays false, preserving current behaviour; deployments whose
queries all name a redis source can turn it on. Note TranQL's own /search
endpoint infers default indexes from the scanned schema (api.py), so that
endpoint does depend on it.

SCHEMA_UPDATE_INTERVAL is exposed at the same time, because 20 minutes is
only safe relative to how long the scan takes and 1200s against an 833s
build is not.

Booleans out of Config need coercing: Config.__getitem__ returns
environment overrides verbatim, so SKIP_REDIS_SCHEMA=false arrives as the
string "false" and bool("false") is True. util.as_bool handles that and
raises on anything it cannot interpret, rather than silently picking a
branch. The existing flags read via Config.get have the same latent issue
if set from the environment; not changed here.

Tests: 22 covering as_bool's string forms, that skip_redis=True does not
construct RedisAdapter while still loading the connection params, and that
skip_redis=False still does scan, so the fix cannot silently disable it for
everyone. tests/test_tranql.py is 8 failed / 41 passed both before and
after this change on origin/develop.
The previous commit defaulted metadata['schema'] so the skip path stopped
raising KeyError, but that only moved the crash. add_layer calls
decorate_schema, which scores edges from adapter.summary and gets the
adapter via RedisAdapter._get_adapter(name). The registration it looks up
happens in set_adapter, inside the branch skip_redis skips, so
_get_adapter raised:

  ValueError: Redis backend with name redis not registered.

Caught against the real schema.yaml on a deployed image, not by the tests:
the fixture in the previous commit had an empty type map, so add_layer
iterated nothing and decorate_schema was never reached. decorate_schema
also returns early unless the entry is named exactly 'redis', which hid it
further.

The summary being decorated exists only because the scan ran, so with the
scan skipped there is nothing to score and leaving the schema undecorated
is the correct outcome, not a workaround. Guarded on the registry rather
than threading skip_redis down, so any caller reaching decorate_schema
without a registered adapter degrades instead of raising.

Test now uses a non-empty layer under an entry named 'redis', which is what
it takes to reach this code, and asserts no adapter is constructed while
the connection params still load.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant