From 7fbf53ee54be04b19d52b9117431fa5ee0df758e Mon Sep 17 00:00:00 2001 From: Maqbool Date: Thu, 29 Oct 2026 22:54:28 +0400 Subject: [PATCH 1/2] Bring the roadmap up to date with phases 2 and 3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records what actually works — connect, authenticate, run any statement returning no rows — with the shape of the calls rather than a claim, and that SELECT is still refused rather than guessed at. Ticks phase 2 up to the point reached and names what is left in it, with why a modern server at default settings cannot be reached until caching_sha2_password lands. Keeps the auth-switch finding, since it is the one correction a real server made to a design the fake had happily confirmed. Also states plainly that CI does not prove the login: the workflow has no MySQL service, so a green check means the parsers agree with the spec and the fixtures, nothing more. --- ROADMAP.md | 81 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 58 insertions(+), 23 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index d2e8783..81bca2f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -16,18 +16,38 @@ Decisions made along the way are recorded in ## Where things stand -- Toolchain current: Erlang 27.3.4.15 / Elixir 1.20.4, `db_connection` 2.10, CI green -- Phase 1 is done: framing, the socket read loop, the wire-type codecs, and the - three generic response packets. 31 tests, 4 of them properties -- `Mysqlex.Protocol` still holds the 2018 handshake parse and the stubbed - `DBConnection` callbacks; nothing routes through `Mysqlex.Client` yet -- No dispatcher: nothing inspects a payload's first byte to choose a parser - -**Known broken, fix first:** `parse_payload_body/2` reads `server_version` as a -fixed 6 bytes, but the protocol says NUL-terminated; `auth_plugin_data_part_2` is -fixed at 13 bytes where the spec says `MAX(13, len - 8)`. Both are in -`lib/mysqlex/protocol.ex` and both are the kind of thing phase 1's codecs make -easy to get right. +**The driver connects to a real MySQL server, authenticates, and runs any +statement that returns no rows.** 58 tests with `--include integration`, 50 and +4 excluded without. + +- Toolchain current: Erlang 27.3.4.15 / Elixir 1.20.4, `db_connection` 2.10 +- Phases 1 and 2 are done; phase 3 has `COM_QUERY` but not resultsets +- No line of the February 2018 code remains. `Mysqlex.Protocol` is deleted; the + `DBConnection` callbacks now live in `Mysqlex.Connection` as a thin wrapper + whose state is a `%Mysqlex.Client{}` +- Both 2018 handshake bugs are fixed, and verified against bytes captured from a + real server: the old pattern raises on MySQL 8.4.4 and parses only when the + version string happens to be six characters, as 5.7.21's was + +What works end to end against MySQL 8.0: + +```elixir +{:ok, client} = Client.connect(hostname: "127.0.0.1", port: 3307, + username: "mysqlex", password: "secret", + database: "mysqlex_test") + +{:ok, insert, client} = Client.query(client, "INSERT INTO widgets (name) VALUES ('first')") +insert.affected_rows # => 1 +insert.last_insert_id # => 1 +``` + +`SELECT` returns `{:error, :resultset_not_implemented, client}` until the +resultset parsers land, which is the next slice. + +**CI does not prove the login.** The workflow has no MySQL service, so it runs +the default suite only. A green check means the parsers agree with the spec and +the fixtures, not that a connection succeeded. Adding a `services: mysql:8.0` +leg would close that gap. ## How to work on this @@ -94,34 +114,49 @@ Each phase should end somewhere runnable. Deliberately not here: -- Sequence id tracking across a command — it is a parameter to the codec, and - tracking belongs to whatever owns the connection, which does not exist yet +- Sequence id tracking across a command — it stays a parameter to the codec. + Done in phase 3, where `Client.query/3` resets it to 0 per command, which is + the first point at which there was a command to track it across - The dispatcher that decides whether a payload beginning `0xFE` is an EOF packet, an OK standing in for one, or a length-encoded integer. It needs the payload length and the command context - `CLIENT_SESSION_TRACK` in the OK packet — the capability will not be advertised until phase 5 -### 2. Connection phase +### 2. Connection phase — logging in works -- [ ] Fix the `HandshakeV10` field parse -- [ ] Capability flag negotiation — decide what the client claims -- [ ] `HandshakeResponse41` -- [ ] `mysql_native_password` (get connected first) -- [ ] `AuthSwitchRequest` / `AuthSwitchResponse` / `AuthMoreData` +- [x] Fix the `HandshakeV10` field parse +- [x] Capability flag negotiation — what the client claims, and what it refuses +- [x] `HandshakeResponse41` +- [x] `mysql_native_password` +- [x] `AuthSwitchRequest` / `AuthSwitchResponse` +- [ ] `AuthMoreData`, needed by `caching_sha2_password` - [ ] `caching_sha2_password` — fast path, then the full RSA public key path - [ ] `sha256_password` - [ ] `SSLRequest` and the TLS upgrade - [ ] Expired passwords, `AuthNextFactor` (MFA) -### 3. Text protocol +Until `caching_sha2_password` lands, a modern server at default settings cannot +be reached — the account has to be created `IDENTIFIED WITH +mysql_native_password`, and MySQL 8.4 ships that plugin disabled. Hence the 8.0 +pin in the integration setup. + +The server names its *default* plugin in the greeting, before it knows which +account is connecting, and settles the real one with an `AuthSwitchRequest` +afterwards. Refusing on the greeting's plugin rejects logins the server would +have completed. This was found by running against a real server; no scripted test +would have shown it, because a fake only replays what you already believe. + +### 3. Text protocol — statements run, rows do not come back yet -- [ ] `COM_QUERY` +- [x] `COM_QUERY`, and the response dispatch on its first byte - [ ] Text resultset: column count, column definitions, rows, `0xFB` for NULL - [ ] Decode every `MYSQL_TYPE_*` to an Elixir term — decimal, date/time/datetime, JSON, bit, blob, the lot - [ ] Multi-resultset via `SERVER_MORE_RESULTS_EXISTS`, and `CLIENT_MULTI_STATEMENTS` -- [ ] `LOCAL INFILE` request/data +- [ ] `LOCAL INFILE` request/data — currently refused outright, since + `CLIENT_LOCAL_FILES` is not claimed and a server asking the client for a + file is either broken or hostile ### 4. Binary protocol and prepared statements From 27b5dfc5e39f131d73f5daa6f17ac87aeb5dd193 Mon Sep 17 00:00:00 2001 From: Maqbool Date: Fri, 30 Oct 2026 14:42:38 +0400 Subject: [PATCH 2/2] Reframe the roadmap around the DDD pivot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a "pivot" section separating what the restructure genuinely buys from what is renaming, and what is deliberately not adopted — no repositories, no domain events, no service layers over one-liners — with the reason in each case. Reorders the phases. The text resultset stays first, because every layer above it needs something to read and because it is the largest body of logic left. Then the restructure, then the query layer, then prepared statements, which is where a query compiled to placeholders becomes executable rather than only inspectable. Records the open question the query layer turns on: placeholders or interpolated literals. Placeholders need prepared statements and are the only version that is not an injection hole, so the intent is placeholders, landing before anything can run them. Drops the Ecto adapter as a destination, since the pivot builds that layer directly. Still possible, no longer the aim. Also states the current CRUD position plainly: three letters of four, with no parameter binding and no escaping helper, which is what the query layer plus prepared statements exist to close. --- ROADMAP.md | 264 +++++++++++++++++++++++------------------- lib/mysqlex/client.ex | 2 +- 2 files changed, 149 insertions(+), 117 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 81bca2f..c67d0db 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,53 +1,106 @@ # Roadmap -Rough plan for implementing the MySQL client/server protocol. Deliberately not a -spec — the order matters more than the detail, and the detail lives in -[`docs/mysql-protocol/`](docs/mysql-protocol/README.md). +Rough plan. Deliberately not a spec — the order matters more than the detail, and +the detail lives in [`docs/mysql-protocol/`](docs/mysql-protocol/README.md). +Decisions and their reasoning are in [`docs/decisions.md`](docs/decisions.md). ## Goal -Implement everything in the protocol that can be implemented, not the minimum -path to a working query. Eventually an Ecto integration on top. Not a published -library — this is a personal project, so protocol completeness and understanding -are the point, and packaging polish is not. +Implement everything in the protocol that can be implemented, and then build the +layers above it rather than borrowing them: -Decisions made along the way are recorded in -[docs/decisions.md](docs/decisions.md), including the ones still open. +- **The whole protocol.** Both text and binary, every auth method, prepared + statements, cursors, transactions, replication. Not the shortest path to a + working query. +- **Its own query layer.** A composable query value and a compiler to SQL, rather + than an Ecto adapter. See "The pivot" below. +- **`db_connection` for pooling only.** It stays for connection lifecycle, + checkout and the crash-safety discipline that is genuinely hard; nothing above + the pool depends on it. -## Where things stand +Not a published library. Protocol completeness and understanding are the point; +packaging polish is not. -**The driver connects to a real MySQL server, authenticates, and runs any -statement that returns no rows.** 58 tests with `--include integration`, 50 and -4 excluded without. +## The pivot + +**2026-10-30.** The project now also aims at a domain-oriented structure and its +own query stack, rather than a driver plus an Ecto adapter. + +What that changes, honestly separated: + +**Genuinely new** + +- An explicit connection state machine, so illegal operations are unrepresentable. + "A resultset is half-read, do not send another command" is a real invariant that + nothing currently enforces, and it stops being theoretical once cursors exist. +- A `query` context: a composable query value plus a compiler to SQL. This is the + substance of the pivot. +- `ssl.ex` behind the transport behaviour, which is also what unblocks + `caching_sha2_password`'s full authentication path. + +**Mostly renaming** + +- `protocol/` becomes `wire/`; `client.ex` becomes `connection/connection.ex`. + Worth doing while the code is small; it buys vocabulary, not correctness. + +**Deliberately not adopted** + +- Repositories. This *is* the persistence mechanism; a repository over packets is + not a thing. +- Domain events. A strict request/response protocol with no subscribers does not + need them, and per-row events in a resultset would be actively harmful. + Telemetry covers the real need. +- Service layers over functions that are already one-liners. + +**Reading of "immutability style":** functional core, imperative shell. Encoding +stays pure functions, IO is confined to the transport, and state transitions +return new values. + +**Open question for review:** does the compiler emit SQL with interpolated +literals, or SQL with placeholders plus a parameter list? Placeholders need +prepared statements to execute, which is more work and the only version that is +not an injection hole. Current intent is placeholders, with the query layer landing +before prepared statements can run it — so it would be compile-and-inspect before +it is compile-and-execute. + +## Target structure -- Toolchain current: Erlang 27.3.4.15 / Elixir 1.20.4, `db_connection` 2.10 -- Phases 1 and 2 are done; phase 3 has `COM_QUERY` but not resultsets -- No line of the February 2018 code remains. `Mysqlex.Protocol` is deleted; the - `DBConnection` callbacks now live in `Mysqlex.Connection` as a thin wrapper - whose state is a `%Mysqlex.Client{}` -- Both 2018 handshake bugs are fixed, and verified against bytes captured from a - real server: the old pattern raises on MySQL 8.4.4 and parses only when the - version string happens to be six characters, as 5.7.21's was - -What works end to end against MySQL 8.0: - -```elixir -{:ok, client} = Client.connect(hostname: "127.0.0.1", port: 3307, - username: "mysqlex", password: "secret", - database: "mysqlex_test") - -{:ok, insert, client} = Client.query(client, "INSERT INTO widgets (name) VALUES ('first')") -insert.affected_rows # => 1 -insert.last_insert_id # => 1 ``` +lib/mysqlex/ +├── wire/ the wire format +│ ├── packet.ex frame +│ ├── codec.ex wire types +│ └── packets/ one module per packet +├── connection/ the conversation +│ ├── connection.ex aggregate root +│ ├── phase.ex state machine +│ └── transport/ tcp, ssl +├── query/ building SQL +│ ├── query.ex compiler.ex +│ └── fragments/ from, where, select, join, order, limit +└── result.ex row.ex error.ex +``` + +## Where things stand -`SELECT` returns `{:error, :resultset_not_implemented, client}` until the -resultset parsers land, which is the next slice. +**The driver connects to a real MySQL server, authenticates, and runs any +statement that returns no rows.** 58 tests with `--include integration`, 50 and 4 +excluded without. -**CI does not prove the login.** The workflow has no MySQL service, so it runs -the default suite only. A green check means the parsers agree with the spec and -the fixtures, not that a connection succeeded. Adding a `services: mysql:8.0` -leg would close that gap. +- Toolchain current: Erlang 27.3.4.15 / Elixir 1.20.4, `db_connection` 2.10 +- No line of the February 2018 code remains +- Both 2018 handshake bugs are fixed, verified against bytes captured from a real + server +- CRUD is three letters of four: `INSERT`, `UPDATE`, `DELETE` and DDL all work; + `SELECT` returns `{:error, :resultset_not_implemented, client}` +- Transaction *statements* already work, since `START TRANSACTION` and `COMMIT` + answer with OK packets. `DBConnection`'s transaction callbacks are still stubs +- No parameter binding and no escaping helper, so values must be interpolated by + hand. That is an injection hole by construction and is what the query layer plus + prepared statements exist to close + +**CI does not prove the login.** The workflow has no MySQL service, so a green +check means the parsers agree with the spec and the fixtures, nothing more. ## How to work on this @@ -100,108 +153,87 @@ GRANT ALL ON mysqlex_test.* TO 'mysqlex'@'%'; ## Phases -Each phase should end somewhere runnable. - -### 1. Foundation — done - -- [x] Frame codec: 3-byte little-endian length + 1-byte sequence id -- [x] Packets larger than 2^24-1: splitting on send, reassembly on receive -- [x] Buffering — a packet can span several TCP reads, or several arrive in one -- [x] `Mysqlex.Transport` behaviour, so reads can be driven without a server -- [x] Codecs: `int`, `string`, `string`; `string` is - `rest` and fixed-width fields are inline patterns, so neither needs a function -- [x] `OK_Packet`, `ERR_Packet`, `EOF_Packet` - -Deliberately not here: - -- Sequence id tracking across a command — it stays a parameter to the codec. - Done in phase 3, where `Client.query/3` resets it to 0 per command, which is - the first point at which there was a command to track it across -- The dispatcher that decides whether a payload beginning `0xFE` is an EOF - packet, an OK standing in for one, or a length-encoded integer. It needs the - payload length and the command context -- `CLIENT_SESSION_TRACK` in the OK packet — the capability will not be - advertised until phase 5 - -### 2. Connection phase — logging in works - -- [x] Fix the `HandshakeV10` field parse -- [x] Capability flag negotiation — what the client claims, and what it refuses -- [x] `HandshakeResponse41` -- [x] `mysql_native_password` -- [x] `AuthSwitchRequest` / `AuthSwitchResponse` -- [ ] `AuthMoreData`, needed by `caching_sha2_password` -- [ ] `caching_sha2_password` — fast path, then the full RSA public key path -- [ ] `sha256_password` -- [ ] `SSLRequest` and the TLS upgrade -- [ ] Expired passwords, `AuthNextFactor` (MFA) +Reordered for the pivot. The first item is unchanged and comes first for a +practical reason: every layer above it needs something to read. + +### A. Finish the text resultset — next + +- [ ] Column count, column definitions, rows, `0xFB` for NULL +- [ ] The `0xFE` disambiguation: EOF packet, an OK standing in for one, or a + length-encoded integer, decided by payload length +- [ ] Replace the catch-all that currently reports `:resultset_not_implemented` + for malformed payloads as well as for resultsets +- [ ] Decode every `MYSQL_TYPE_*` to an Elixir term — decimal, the date/time + family, JSON, bit, blob, unsigned 64-bit, `TINYINT(1)` + +This is the largest body of logic in the project and the highest value per hour +on the list: it is what turns a write-only driver into one you can query with. -Until `caching_sha2_password` lands, a modern server at default settings cannot -be reached — the account has to be created `IDENTIFIED WITH -mysql_native_password`, and MySQL 8.4 ships that plugin disabled. Hence the 8.0 -pin in the integration setup. +### B. Restructure -The server names its *default* plugin in the greeting, before it knows which -account is connecting, and settles the real one with an `AuthSwitchRequest` -afterwards. Refusing on the greeting's plugin rejects logins the server would -have completed. This was found by running against a real server; no scripted test -would have shown it, because a fake only replays what you already believe. +- [ ] `wire/` and `wire/packets/` +- [ ] `connection/` with `Phase` as an explicit state machine +- [ ] `transport/` as the anti-corruption boundary +- [ ] `Result` and `Row` value objects -### 3. Text protocol — statements run, rows do not come back yet +### C. The query layer -- [x] `COM_QUERY`, and the response dispatch on its first byte -- [ ] Text resultset: column count, column definitions, rows, `0xFB` for NULL -- [ ] Decode every `MYSQL_TYPE_*` to an Elixir term — decimal, date/time/datetime, - JSON, bit, blob, the lot -- [ ] Multi-resultset via `SERVER_MORE_RESULTS_EXISTS`, and `CLIENT_MULTI_STATEMENTS` -- [ ] `LOCAL INFILE` request/data — currently refused outright, since - `CLIENT_LOCAL_FILES` is not claimed and a server asking the client for a - file is either broken or hostile +- [ ] A composable query value +- [ ] Fragments: from, select, where, join, order, limit, offset +- [ ] A compiler to SQL plus parameters +- [ ] Decide and record: placeholders or interpolation -### 4. Binary protocol and prepared statements +### D. Prepared statements -- [ ] `COM_STMT_PREPARE` and its response (param + column definitions) +- [ ] `COM_STMT_PREPARE` and its response - [ ] `COM_STMT_EXECUTE`: parameter binding, type encoding, null bitmap - [ ] Binary resultset rows - [ ] `COM_STMT_SEND_LONG_DATA`, `COM_STMT_RESET`, `COM_STMT_CLOSE` - [ ] Cursors: `CURSOR_TYPE_READ_ONLY` + `COM_STMT_FETCH` -- [ ] Decide: are prepared statements cached per connection? +- [ ] Decide: cached per connection? -### 5. Session and utility commands +Where the query layer becomes executable rather than inspectable. -- [ ] `COM_PING`, `COM_QUIT`, `COM_INIT_DB`, `COM_RESET_CONNECTION` -- [ ] `COM_CHANGE_USER`, `COM_SET_OPTION`, `COM_STATISTICS`, `COM_DEBUG`, `COM_FIELD_LIST` -- [ ] Session state tracking (`SESSION_TRACK_*` in the OK packet) -- [ ] Compression: zlib, then zstd +### E. Authentication and TLS -### 6. Complete the DBConnection behaviour +- [ ] `AuthMoreData` +- [ ] `caching_sha2_password` — fast path, then full authentication +- [ ] `ssl.ex` behind the transport behaviour +- [ ] `sha256_password` +- [ ] Expired passwords, `AuthNextFactor` (MFA) -Each stub in `lib/mysqlex/protocol.ex` maps onto protocol work above: +Can float earlier. Until it lands, only accounts created `IDENTIFIED WITH +mysql_native_password` are reachable, and MySQL 8.4 ships that plugin disabled — +hence the 8.0 pin in the integration setup. -| Callback | Implemented by | -| --- | --- | -| `handle_prepare` / `handle_execute` / `handle_close` | phase 4 | -| `handle_declare` / `handle_fetch` / `handle_deallocate` | phase 4 cursors (`Repo.stream`) | -| `handle_begin` / `handle_commit` / `handle_rollback` | transactions + savepoints | -| `ping` / `disconnect` | `COM_PING` / `COM_QUIT` | +### F. Reduce `db_connection` to pooling - [ ] Transactions and savepoints, `handle_status` -- [ ] Replace the remaining stubs +- [ ] Replace the remaining stubs so `DBConnection.execute/3` works +- [ ] Keep nothing above the pool dependent on it + +### G. Session and utility commands -Replacing `db_connection` with a hand-written connection and pool is an option -from here, not a prerequisite — see decision 15. +- [ ] `COM_PING`, `COM_QUIT`, `COM_INIT_DB`, `COM_RESET_CONNECTION` +- [ ] `COM_CHANGE_USER`, `COM_SET_OPTION`, `COM_STATISTICS`, `COM_DEBUG`, + `COM_FIELD_LIST` +- [ ] Session state tracking (`SESSION_TRACK_*`) +- [ ] Multi-resultset via `SERVER_MORE_RESULTS_EXISTS` +- [ ] `LOCAL INFILE` — currently refused outright, since `CLIENT_LOCAL_FILES` is + not claimed and a server asking the client for a file is either broken or + hostile +- [ ] Compression: zlib, then zstd -### 7. Replication +### H. Replication - [ ] `COM_BINLOG_DUMP` - [ ] Binlog event header and event parsing -### Later: Ecto +### Dropped from the plan -Out of scope for now, and worth knowing it is two jobs. The driver is this -project; an `Ecto.Adapters.SQL` adapter also needs a SQL-generation module that -turns Ecto queries into MySQL SQL. That half is query compilation rather than -protocol, and is comparable in size. +An `Ecto.Adapters.SQL` adapter. The pivot builds the query layer directly, so the +adapter is no longer the destination. It stays possible — the driver would still +satisfy one — but it is not what this is aiming at. ## References diff --git a/lib/mysqlex/client.ex b/lib/mysqlex/client.ex index fc0a161..d1bea70 100644 --- a/lib/mysqlex/client.ex +++ b/lib/mysqlex/client.ex @@ -41,7 +41,7 @@ defmodule Mysqlex.Client do def handshake(transport, socket, opts) do timeout = Keyword.get(opts, :connect_timeout, @default_timeout) username = Keyword.fetch!(opts, :username) - password = Keyword.get(opts, :password) || "" + password = Keyword.get(opts, :password) database = Keyword.get(opts, :database) client = %__MODULE__{