Problem
Database::Fetch builds a FetchQueryResults whose row_values is a std::vector<std::vector<std::wstring>> (include/fetch_query_results.h). FetchRecordsQuery::GetResults stringifies every column into a wstring via GetColumnValue, and FetchRecordsQuery::Hydrate then parses each string back into the typed member. This text round-trip on the read path is both a performance cost and the root cause of two concrete data-corruption bugs.
Concrete defects caused by the text round-trip
1. 64-bit integers truncated to 32 bits (consolidated from #15)
GetColumnValue reads SQLITE_INTEGER with the 32-bit sqlite3_column_int (return std::to_wstring(sqlite3_column_int(stmt_, col));), and FetchMaxIdQuery::GetMaxId does the same. Any stored int64_t above INT32_MAX — including AUTOINCREMENT ids once they grow large — is silently truncated/wrapped on read, breaking subsequent Fetch(id) / Update / Delete. Fix: use sqlite3_column_int64.
2. REAL values lose precision (consolidated from #16)
GetColumnValue formats SQLITE_FLOAT via std::to_wstring(double), which uses %f with a fixed 6 decimal places, so e.g. 0.123456789 round-trips back as 0.123457 and large/scientific magnitudes are mangled. Writes are fine (sqlite3_bind_double stores the true value); the loss is only on the read path. Fix: read sqlite3_column_double directly, or format with full round-trip precision (%.17g).
Performance / memory
- Every value incurs an allocation + parse on top of the SQLite copy; significant for wide rows / large result sets.
- The entire result set is materialized as strings before hydration (no streaming), so large queries hold the whole set twice (string + typed form).
Suggested direction
Hydrate directly from the prepared statement using the typed column accessors (sqlite3_column_int64, sqlite3_column_double, sqlite3_column_text/_bytes) into the destination members, skipping the intermediate wstring. This removes the per-value allocation/parse, fixes both data-corruption bugs above at the source, and opens the door to row-by-row streaming.
Acceptance criteria
- Integer columns round-trip values greater than
INT32_MAX exactly (regression test).
- Double columns round-trip high-precision values exactly (regression test).
- Hydration reads typed columns directly rather than via
std::wstring.
Related (tracked separately)
Consolidates #15 (int truncation) and #16 (REAL precision), which were symptoms of this single root cause and are closed as duplicates of this issue.
Problem
Database::Fetchbuilds aFetchQueryResultswhoserow_valuesis astd::vector<std::vector<std::wstring>>(include/fetch_query_results.h).FetchRecordsQuery::GetResultsstringifies every column into awstringviaGetColumnValue, andFetchRecordsQuery::Hydratethen parses each string back into the typed member. This text round-trip on the read path is both a performance cost and the root cause of two concrete data-corruption bugs.Concrete defects caused by the text round-trip
1. 64-bit integers truncated to 32 bits (consolidated from #15)
GetColumnValuereadsSQLITE_INTEGERwith the 32-bitsqlite3_column_int(return std::to_wstring(sqlite3_column_int(stmt_, col));), andFetchMaxIdQuery::GetMaxIddoes the same. Any storedint64_taboveINT32_MAX— includingAUTOINCREMENTids once they grow large — is silently truncated/wrapped on read, breaking subsequentFetch(id)/Update/Delete. Fix: usesqlite3_column_int64.2. REAL values lose precision (consolidated from #16)
GetColumnValueformatsSQLITE_FLOATviastd::to_wstring(double), which uses%fwith a fixed 6 decimal places, so e.g.0.123456789round-trips back as0.123457and large/scientific magnitudes are mangled. Writes are fine (sqlite3_bind_doublestores the true value); the loss is only on the read path. Fix: readsqlite3_column_doubledirectly, or format with full round-trip precision (%.17g).Performance / memory
Suggested direction
Hydrate directly from the prepared statement using the typed column accessors (
sqlite3_column_int64,sqlite3_column_double,sqlite3_column_text/_bytes) into the destination members, skipping the intermediatewstring. This removes the per-value allocation/parse, fixes both data-corruption bugs above at the source, and opens the door to row-by-row streaming.Acceptance criteria
INT32_MAXexactly (regression test).std::wstring.Related (tracked separately)
StringUtilities::ToInt/ToDoublesilently return0on parse failure. This is an independent error-handling defect in those helpers; moving off the text path (per this redesign) reduces reliance on them but does not by itself fix them, so it remains its own issue.Consolidates #15 (int truncation) and #16 (REAL precision), which were symptoms of this single root cause and are closed as duplicates of this issue.