Repository navigation
feat(ingest): add hotdata ingest raw-sql verb
#241
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -268,6 +268,19 @@ impl IngestClient { | |
| ) | ||
| } | ||
|
|
||
| /// Ingest the result of a source-native SQL query (SQL datasources only). | ||
| /// The SQL runs verbatim in the source dialect; the server enforces a single | ||
| /// read-only statement. | ||
| pub fn create_raw_query(&self, req: &RawSqlIngest) -> Result<IngestAck, IngestError> { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: (not blocking) No test for this endpoint. The module's mockito suite pins the wire contract for the other enqueue routes ( super nit: (not blocking) the |
||
| self.require_api_key()?; | ||
| let body = serde_json::to_value(req).expect("RawSqlIngest serializes"); | ||
| self.send( | ||
| self.authed(reqwest::Method::POST, "/queries/raw") | ||
| .json(&body), | ||
| Some(&body), | ||
| ) | ||
| } | ||
|
|
||
| pub fn drain(&self) -> Result<serde_json::Value, IngestError> { | ||
| self.send(self.authed(reqwest::Method::POST, "/jobs/drain"), None) | ||
| } | ||
|
|
@@ -429,6 +442,18 @@ pub struct QueryToIngest { | |
| pub database_id: Option<String>, | ||
| } | ||
|
|
||
| /// `POST /ingest/queries/raw` body: source-native SQL run verbatim against a SQL | ||
| /// datasource, its result loaded into `table`. The workspace rides the | ||
| /// X-Workspace-Id header, so it is not in the body. | ||
| #[derive(Debug, Serialize)] | ||
| pub struct RawSqlIngest { | ||
| pub sql: String, | ||
| pub source: String, | ||
| pub table: String, | ||
| #[serde(skip_serializing_if = "Option::is_none")] | ||
| pub limit: Option<u64>, | ||
| } | ||
|
|
||
| /// 202 body from `POST /sources` and `POST /queries`. Common fields are typed; | ||
| /// everything else (e.g. `resource`, `columns`, `where`, `limit` on queries) is | ||
| /// captured for `--output json`. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -37,7 +37,7 @@ | |
| //! `name`, `type`, `status`, `stage`, …), not the wire rows. | ||
|
|
||
| use crate::client::ingest::{ | ||
| ConnectorEntry, IngestAck, IngestClient, IngestRequest, JobStatus, QueryToIngest, | ||
| ConnectorEntry, IngestAck, IngestClient, IngestRequest, JobStatus, QueryToIngest, RawSqlIngest, | ||
| }; | ||
| use crate::commands::prompt::{ | ||
| ask_secret, ask_text, optional, optional_default, prompt_list, select_optional, | ||
|
|
@@ -160,6 +160,34 @@ pub enum IngestCommands { | |
| poll: PollArgs, | ||
| }, | ||
|
|
||
| /// Ingest the result of a source-native SQL query (SQL datasources only) | ||
| /// | ||
| /// Unlike `new-import` (SELECT <cols|*> FROM <ds>[.<table>] [WHERE] [LIMIT]), | ||
| /// the SQL runs verbatim against the source engine in its OWN dialect — | ||
| /// joins, aggregation, GROUP BY, CTEs, window functions, and engine-specific | ||
| /// functions all execute at the source; only the result set transfers. It | ||
| /// must be a single read-only statement; the result lands in `--table` in a | ||
| /// fresh managed database. | ||
| RawSql { | ||
| /// A single read-only SQL statement, in the SOURCE engine's dialect | ||
| sql: String, | ||
|
|
||
| /// SQL datasource to run against: a name or a datasource id | ||
| #[arg(long)] | ||
| source: String, | ||
|
|
||
| /// Result table name in the new managed database | ||
| #[arg(long)] | ||
| table: String, | ||
|
|
||
| /// Cap the result at N rows | ||
| #[arg(long)] | ||
| limit: Option<u64>, | ||
|
|
||
| #[command(flatten)] | ||
| poll: PollArgs, | ||
| }, | ||
|
|
||
| /// List your imports: the SQL behind each and the database it landed in | ||
| ListImports, | ||
|
|
||
|
|
@@ -252,6 +280,13 @@ pub fn dispatch(workspace_id: &str, output: &str, command: IngestCommands) { | |
| database_id, | ||
| poll, | ||
| } => new_import(workspace_id, output, sql, all, source, database_id, &poll), | ||
| IngestCommands::RawSql { | ||
| sql, | ||
| source, | ||
| table, | ||
| limit, | ||
| poll, | ||
| } => raw_sql(workspace_id, output, sql, source, table, limit, &poll), | ||
| IngestCommands::ListImports => list_imports(workspace_id, output), | ||
| IngestCommands::TriggerImport { id, poll } => { | ||
| trigger_import(workspace_id, output, &id, &poll) | ||
|
|
@@ -940,6 +975,48 @@ fn looks_like_ingest_id(s: &str) -> bool { | |
| s.len() == 32 && s.chars().all(|c| c.is_ascii_hexdigit()) | ||
| } | ||
|
|
||
| // --- raw-sql: source-native SQL front-door -------------------------------- | ||
|
|
||
| fn raw_sql( | ||
| workspace_id: &str, | ||
| output: &str, | ||
| sql: String, | ||
| source: String, | ||
| table: String, | ||
| limit: Option<u64>, | ||
| poll: &PollArgs, | ||
| ) { | ||
| let client = IngestClient::new(workspace_id); | ||
| let req = RawSqlIngest { | ||
| sql, | ||
| source, | ||
| table, | ||
| limit, | ||
| }; | ||
| let spinner = util::spinner("submitting raw-sql import…"); | ||
| let ack = client.create_raw_query(&req).unwrap_or_else(|e| { | ||
| spinner.finish_and_clear(); | ||
| e.exit() | ||
| }); | ||
|
Comment on lines
+996
to
+1000
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: (not blocking) super nit: (not blocking) the tail of this fn (spinner → |
||
| spinner.finish_and_clear(); | ||
| if poll.wait { | ||
| let st = poll_ingest( | ||
| &client, | ||
| output, | ||
| &ack.ingest_id, | ||
| poll.wait_timeout, | ||
| "importing", | ||
| true, | ||
| ); | ||
| render_done(&st, output); | ||
| return; | ||
| } | ||
| // Non-blocking default: kick the drain so the enqueued job actually runs, | ||
| // then hand the terminal back (same as new-import). | ||
| let _ = client.drain(); | ||
| render_ack(&ack, "import id:", output); | ||
| } | ||
|
|
||
| /// The SQL a `new-import` runs: explicit SQL wins; `--all` becomes a full | ||
| /// `SELECT *` against the datasource's name (`pinned_name` when --source was | ||
| /// a datasource id). Errors are messages for `fail`. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: (not blocking)
create_raw_querycallsrequire_api_key(), soraw-sqlis an enqueue verb — but the auth paragraph atskills/hotdata/SKILL.md:238still lists onlynew-datasource,new-import,trigger-import,delete-datasourceas needing a workspace API key. An agent reading that line will assume a login session works forraw-sqland hitNeedsApiKey. Worth addingraw-sqlto that list.Also,
--limitisn't mentioned anywhere in the skill doc — the example only shows--source/--table.