-
Notifications
You must be signed in to change notification settings - Fork 105
fix: batch domain-label inserts under one stable query per label #499
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -120,11 +120,16 @@ def insert_domain_entity(entity:Entity): | |
| if entity.classification and entity.classification == LOCAL_ENTITY_CLASSIFICATION: | ||
| return | ||
|
|
||
| e_var = new_query_var() | ||
| e_id = entity.entityId | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: |
||
| e_label = escape_cypher_label(label_from(entity.classification or DEFAULT_CLASSIFICATION)) | ||
| e_comment = f'// awsqid:{e_id}-{e_label}'.replace('\r', ' ').replace('\n', ' ') | ||
| query_e = f"UNWIND $params AS params MERGE ({e_var}:`__Entity__`{{{graph_client.node_id('entityId')}: params.entityId}}) SET {e_var} :`{e_label}` {e_comment}" | ||
| # The query text must be stable for a given label: the batch | ||
| # client groups param rows by the full query string, so a | ||
| # per-entity variable name or a per-entity id embedded in the | ||
| # text makes every insert its own single-row batch and defeats | ||
| # batching and dedup entirely (#477). The entity id already | ||
| # travels in the params; the comment stays label-scoped. | ||
| e_comment = f'// awsqid:domain-label-{e_label}'.replace('\r', ' ').replace('\n', ' ') | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Dropping the per-entity id is safe. The only consumer of |
||
| query_e = f"UNWIND $params AS params MERGE (e:`__Entity__`{{{graph_client.node_id('entityId')}: params.entityId}}) SET e :`{e_label}` {e_comment}" | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed the differential locally rather than reading it off the diff. Against current main's builder the new test fails with three separate query keys; on this branch both pass: Build test dir is 27 failed / 15 errors on both this branch and
|
||
| graph_client.execute_query_with_retry(query_e, self._to_params({'entityId': e_id}), max_attempts=5, max_wait=7) | ||
|
|
||
| insert_domain_entity(fact.subject) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rebase note: this predates
dc6a096f(downmerge #502), which already landed #476'sUNWIND $params/_to_paramsconversion on main, hence the conflict. After a rebase the change reduces to the fixedevar plus the label-scoped comment. Main also carries a comment above this block calling the batching "a future optimization"; that should go with the fix.