From 75f8b94ba2bee5adb0c540d18dc236fbb50a10cb Mon Sep 17 00:00:00 2001 From: James Foster Date: Thu, 27 Aug 2026 14:47:51 -0700 Subject: [PATCH] Start notebooks with import gemdb, and fix three things the CLI got wrong Four changes a user ran into in one sitting. **The sample code says `import gemdb`.** A new notebook's first cell, the walkthrough and the README all opened with `import gemstone` -- Grail's own lower-level surface, inherited from Jasper. `gemdb` is the module a developer should meet first: `gemdb.root` for the data that outlives the session, `gemdb.commit()` for the moment it becomes everyone's. **`gemdb` is on the PATH of terminals opened in VS Code.** The command is generated into `~/GemDB/bin`, which is on nobody's PATH, so `which gemdb` answered "not found" and the README's answer was a line the user had to add to their own shell profile. VS Code contributes this per extension and reverts it when the extension is disabled, which keeps it on the automated side of the line; editing a shell profile would not be, so that stays an ask. Cleared before every prepend, because the collection is persisted across window reloads and re-applied before activation. **topaz no longer comments on an exit it was always going to ignore.** `gemdb file.py` from a real terminal ended with four lines about ignoring an EXIT and a `Logging out session 1.`. `topaz -h` says of `-S` that topaz "exits when the script completes" and that "exit and quit commands are ignored" -- silently on a pipe, which is every CI run and both suites, and out loud on a tty. Exit codes never travelled through that line and still do not: sys.exit(3) -> 3, an uncaught exception -> 1, a missing file -> 2, measured again over a pty without it. **Non-ASCII survives `gemdb file.py` in both directions.** The console this mode installs is a GsFile, which takes BYTES: print() wrote a Unicode string's code units, so a bullet in an ASCII-art rabbit came out as binary. The #GrailConsole box gains a second slot naming what the sink takes, because the sink cannot be asked -- the shell's is a ClientForwarder, and any send to one forwards to the client as error 2336, uncatchable in the gem. Needs the matching Grail change (GemTalk/Grail#701), which also decodes the mirror-image fault on the read side: a line typed at the terminal arrived one character per byte. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 35 ++++++++++++++++ CLAUDE.md | 36 +++++++++++++++- README.md | 17 +++++--- docs/demo-rabbit-in-the-hat.md | 5 ++- resources/walkthrough/notebook.md | 4 +- resources/walkthrough/repl.md | 8 ++-- src/__integration__/cli.test.ts | 22 ++++++++++ src/__tests__/cli.test.ts | 70 ++++++++++++++++++++++++++++++- src/cli.ts | 60 ++++++++++++++++++++++++-- src/extension.ts | 12 ++++++ src/notebook.ts | 17 ++++---- 11 files changed, 261 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a7d9666..683346b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,8 +7,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- **The sample code now starts with `import gemdb`.** A new notebook's first + cell, the walkthrough and the README all opened with `import gemstone` — + Grail's own lower-level surface, inherited from Jasper. The first thing a + developer meets is now `gemdb`: `gemdb.root` for the data that outlives the + session, `gemdb.commit()` for the moment it becomes everyone's. + +### Added + +- **`gemdb` is on the PATH of terminals you open in VS Code.** The command is + generated into `~/GemDB/bin`, which is on nobody's PATH, so `which gemdb` + answered "not found" and the README's answer was a line you had to add to + your own shell profile. GemDB now contributes that directory to the + terminals this editor launches — and takes it back when the extension is + disabled or uninstalled. Your shell profile is still yours; add it there + too if you want `gemdb` in terminals outside VS Code. + ### Fixed +- **Non-ASCII text now survives `gemdb file.py` in both directions.** + `print()` from a script wrote its characters as UTF-16 code units — a NUL + between every ASCII letter, and anything above U+00FF truncated to one byte + — so a `•` in an ASCII-art rabbit turned the whole drawing into binary. + `input()` had the mirror-image fault: a line typed at the terminal arrived + one character per *byte*, so `wörld` came back six characters long and + mojibake. Only this mode was affected; the GemDB Shell and notebooks were + always right, because they exchange characters over the client connection + rather than bytes through a file. Needs the matching Grail change + (GemTalk/Grail, console writes encode for a byte sink). +- **`gemdb file.py` no longer prints topaz's own commentary.** Running a + script from a real terminal ended with four lines about topaz ignoring an + EXIT command, followed by `Logging out session 1.`. The driver ended with + an `exit` that topaz documents as ignored for the way GemDB runs it — it + already exits when the script completes — and ignoring it is silent on a + pipe but spoken aloud on a terminal, which is why it never showed in CI. + Exit codes are unaffected: they never travelled through that line. - **The GemDB Shell no longer prints the database client's own chatter.** Opening a shell wrote a line like `gcits login: session 0x… lgc 0x… rpc gem processId 4726` onto your terminal diff --git a/CLAUDE.md b/CLAUDE.md index 1dda2fa..e6a35f0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -56,6 +56,17 @@ reversible; ask about what is persistent or global.** unraised it stands down and leaves that to `ensureRunning`, where a `sudo` dialog has a visible cause. +- Putting `gemdb` on the PATH of terminals VS Code opens — + `putCliOnPath` in `cli.ts`, applied to + `context.environmentVariableCollection`. Automated because VS Code owns the + reversal: the entry applies only to terminals this editor launches and goes + away when the extension is disabled. **Editing the user's shell profile + would be the other side of the line** — persistent, global, not ours to + undo — so the README asks rather than does. `clear()` before every + `prepend` because the collection is persisted across window reloads and + re-applied before activation; without it a reload stacks a second entry and + a changed root path leaves the old one in front. + `ensureRunning` in `lifecycle.ts` is the single path to a running database, whether the user pressed Start or just ran a notebook cell. It finishes any outstanding preparation, prompts for shared memory, starts the processes, and @@ -218,7 +229,13 @@ re-running `bundle:grail` on every supported platform. **The CLI's exit codes go through a status file, not topaz.** topaz cannot carry an exit status out of a `run` block — `ExitClientError status:` is not -translated, and an `iferr … exit 1` action exits 0 (all measured). So +translated, and an `iferr … exit 1` action exits 0 (all measured). The driver +therefore ends with no `exit` command at all, and must not grow one: `topaz -h` +says of `-S` that topaz "exits when the script completes" and that "exit and +quit commands are ignored". Ignored silently when stdin is a pipe — which is +every CI run, every test, and every `gemdb x.py | cat` — and out loud when +stdin is a tty, where it printed four lines of explanation and a +`Logging out session 1.` in front of the user. So `gemdb-run.tpz` writes the status to the file named in `GEMDB_STATUS_FILE` and the bash wrapper becomes the exit code. Errors there are caught as `AbstractException`, not `Error`: Grail's Python exceptions live outside the @@ -294,6 +311,23 @@ executeAsync loop clears the stack at the next forwarder stop and throws `Error: KeyboardInterrupt - `. Anything new that evaluates Python should go through that layer, not `execute` directly. +**The console box says what the sink takes, because the sink cannot be +asked.** `SessionTemps #GrailConsole` holds an Array; slot 1 is the sink, and +slot 2 — `#'utf8'` — declares that it takes bytes. `gemdb-run.tpz` sets it, +because its sink is `GsFile stdout` and `nextPutAll:` writes a Unicode string's +code units straight through: `print('café')` was UTF-16BE on the terminal, a +NUL between every ASCII character, while the same print through the shell was +right. Grail encodes with `nextPutAsUtf8:` when the slot says so. It cannot +instead probe the sink: the shell installs a `ClientForwarder`, and *any* send +to one — `class`, `respondsTo:`, `isNil` — forwards to the client as error +2336, which is not a Smalltalk exception and is not catchable in the gem +(measured; `on: AbstractException` around `forwarder class` does not run). A +probe would turn every print in a streaming session into a spurious client +stop. The read side is the same seam from the other direction: `GsFile stdin` +answers bytes, so Grail decodes that one branch with `decodeFromUTF8`, keeping +the raw line when it is not UTF-8. Both directions are pinned in +`src/__integration__/cli.test.ts`. + **Stage Grail before stamping it, and stamp only what this run created.** `stageAndRecordGrail` in `grail.ts` owns that order. Reversed, it broke both ways at once: on a first install `/grail` does not exist yet, so diff --git a/README.md b/README.md index b753541..71dfaff 100644 --- a/README.md +++ b/README.md @@ -13,10 +13,10 @@ whose objects *are* the database's objects. Assign to a variable, commit, and it is still there tomorrow. ```python -import gemstone +import gemdb -gemstone["routes"] = load_routes() -gemstone.system.commit() +gemdb.root["routes"] = load_routes() +gemdb.commit() ``` ## Getting started @@ -63,14 +63,21 @@ Setup also writes a shell command to `~/GemDB/bin/gemdb` that behaves like CPython's command line, backed by the database: ```sh -export PATH="$HOME/GemDB/bin:$PATH" # once, in your shell profile - gemdb hello.py # like python3 hello.py gemdb -m some.module # like python3 -m gemdb -c 'print(1+1)' # like python3 -c gemdb # the GemDB Shell ``` +Terminals you open in VS Code already have it: GemDB adds `~/GemDB/bin` to +their PATH, and removes it again if you disable the extension. Terminals +outside VS Code are yours, so GemDB does not edit your shell profile — add it +there yourself if you want `gemdb` everywhere: + +```sh +export PATH="$HOME/GemDB/bin:$PATH" # once, in your shell profile +``` + It needs no environment set up — the wrapper carries its own — and if the database is not running it starts it, the same judgement the editor makes. Exit codes work the way scripts expect: 0 on success, 1 on an uncaught diff --git a/docs/demo-rabbit-in-the-hat.md b/docs/demo-rabbit-in-the-hat.md index 233d858..e71a7bb 100644 --- a/docs/demo-rabbit-in-the-hat.md +++ b/docs/demo-rabbit-in-the-hat.md @@ -8,10 +8,11 @@ below was run against a real database on 2026-08-23; where something surprised me, that is noted rather than tidied away. > **Before you start.** GemDB must be installed (open VS Code once and let it -> finish setting up), and `gemdb` must be on your PATH: +> finish setting up). A terminal opened in VS Code has `gemdb` on its PATH +> already; in any other terminal, put it there: > > ```sh -> export PATH="$HOME/GemDB/bin:$PATH" +> export PATH="$HOME/GemDB/bin:$PATH" # not needed in a VS Code terminal > cd docs/demo > ``` > diff --git a/resources/walkthrough/notebook.md b/resources/walkthrough/notebook.md index 37b4657..c7dd229 100644 --- a/resources/walkthrough/notebook.md +++ b/resources/walkthrough/notebook.md @@ -4,8 +4,8 @@ Open a notebook and pick **GemDB** as the kernel. Cells run inside the database and share variables the way you would expect: ```python -import gemstone -routes = gemstone["routes"] # data already in the database +import gemdb +routes = gemdb.root["routes"] # data already in the database len(routes) ``` diff --git a/resources/walkthrough/repl.md b/resources/walkthrough/repl.md index e3fa7a5..4c1d6df 100644 --- a/resources/walkthrough/repl.md +++ b/resources/walkthrough/repl.md @@ -3,10 +3,10 @@ Opens a Python prompt that runs *inside* the database: ``` ->>> import gemstone ->>> gemstone["answer"] = 42 ->>> gemstone.system.commit() ->>> gemstone["answer"] +>>> import gemdb +>>> gemdb.root["answer"] = 42 +>>> gemdb.commit() +>>> gemdb.root["answer"] 42 ``` diff --git a/src/__integration__/cli.test.ts b/src/__integration__/cli.test.ts index 4c35b08..8fbbabb 100644 --- a/src/__integration__/cli.test.ts +++ b/src/__integration__/cli.test.ts @@ -145,6 +145,28 @@ describe.skipIf(!ready || !canMakeFixture())('the gemdb command', () => { expect(ran.code).toBe(0); }); + it('writes non-ASCII output as UTF-8, not UTF-16 code units', async () => { + // The console this mode installs is a GsFile, which takes BYTES: writing + // a Unicode string to it straight put a NUL between every ASCII character + // and truncated anything above U+00FF to its low byte, so an ASCII-art + // rabbit with a bullet in it came out as binary. Grail encodes because + // the driver's #GrailConsole box declares the sink takes utf8. + const ran = await gemdb('-c', 'print("caf\u00e9 \u2022 \u4e2d")'); + expect(ran.stdout).toContain('caf\u00e9 \u2022 \u4e2d'); + expect(ran.stdout).not.toContain('\u0000'); + expect(ran.code).toBe(0); + }); + + it('reads non-ASCII input as characters, not bytes', async () => { + // The other half of the same seam: GsFile stdin answers bytes, so a + // terminal's UTF-8 arrived one Character per byte — 'w\u00f6rld' as + // 'w\u00c3\u00b6rld', six characters long, and mojibake on the way back + // out. + const ran = await run(['-c', 'n = input()\nprint(n, len(n))'], 'w\u00f6rld\n'); + expect(ran.stdout).toContain('w\u00f6rld 5'); + expect(ran.code).toBe(0); + }); + it('raises EOFError when stdin runs dry', async () => { const ran = await run(['-c', 'input()'], ''); expect(ran.stderr).toContain('EOF'); diff --git a/src/__tests__/cli.test.ts b/src/__tests__/cli.test.ts index d054655..f74e6a7 100644 --- a/src/__tests__/cli.test.ts +++ b/src/__tests__/cli.test.ts @@ -3,7 +3,7 @@ import * as os from 'os'; import * as path from 'path'; import { afterEach, beforeEach, describe, expect, it } from 'vitest'; import { __setSetting } from '../__mocks__/vscode'; -import { cliPath, ensureCliCurrent, writeCliScripts } from '../cli'; +import { cliPath, ensureCliCurrent, putCliOnPath, writeCliScripts } from '../cli'; import { cliStampPath, expectedEnginePath } from '../paths'; /** @@ -78,6 +78,16 @@ describe('writeCliScripts', () => { // A file run is linked topaz, so it never reaches session.ts and would // otherwise sit in the shared cache as the stock 'TopazL'. The truncation // is not optional: 32 characters raises OutOfRange, at login. + // No `exit` command: with -S, topaz exits when the script completes and + // ignores exit outright — quietly on a pipe, but on a tty it prints four + // lines about ignoring it and a spurious "Logging out session 1.". A test + // here because a pipe is what both suites use, so nothing else would + // notice it coming back. + expect(run.split('\n').some((line) => /^\s*(exit|quit)\b/.test(line))).toBe(false); + // The console sink is a GsFile, which takes bytes; the second slot of + // the #GrailConsole box is what tells Grail to encode. Without it, + // non-ASCII print() output is UTF-16 code units on the terminal. + expect(run).toContain("put: (Array with: GsFile stdout with: #'utf8')"); expect(run).toContain('System cacheName: label'); expect(run).toContain("label := 'GemDB run ', label"); expect(run).toContain('label size > 31 ifTrue:'); @@ -199,3 +209,61 @@ describe('writeCliScripts', () => { expect(() => writeCliScripts(ext)).toThrow(/not installed/); }); }); + +describe('putCliOnPath', () => { + /** A stand-in for the collection VS Code applies to its own terminals. */ + function fakeEnvironment(): { cleared: number; prepended: [string, string][] } & { + clear(): void; + prepend(variable: string, value: string): void; + } { + return { + cleared: 0, + prepended: [], + clear() { + this.cleared += 1; + }, + prepend(variable: string, value: string) { + this.prepended.push([variable, value]); + }, + }; + } + + it('prepends the directory the command is generated into', () => { + const env = fakeEnvironment(); + + putCliOnPath(env); + + // Prepended, not appended: a `gemdb` of GemDB's own is the one this + // installation generated. And the trailing delimiter is the whole + // mechanism — without it the entry runs into the rest of the PATH. + expect(env.prepended).toEqual([['PATH', `${path.join(root, 'bin')}${path.delimiter}`]]); + }); + + it('clears first, because the collection outlives the window', () => { + // VS Code persists it across reloads and re-applies it before activation. + // Re-applying without clearing would stack an entry per reload, and a + // changed root path would leave the old one in front of the new. + const env = fakeEnvironment(); + + putCliOnPath(env); + __setSetting('gemdb.rootPath', path.join(root, 'elsewhere')); + putCliOnPath(env); + + expect(env.cleared).toBe(2); + expect(env.prepended[1]).toEqual([ + 'PATH', + `${path.join(root, 'elsewhere', 'bin')}${path.delimiter}`, + ]); + }); + + it('contributes the directory even before the command is written', () => { + // First run: the PATH is set at activation and setup writes bin/gemdb + // minutes later. A terminal opened in between must not need reopening. + fs.rmSync(path.join(root, 'bin'), { recursive: true, force: true }); + const env = fakeEnvironment(); + + putCliOnPath(env); + + expect(env.prepended).toHaveLength(1); + }); +}); diff --git a/src/cli.ts b/src/cli.ts index 247337b..e2cbee3 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -130,6 +130,39 @@ function cliIsCurrent(fingerprint: string): boolean { } } +/** + * The slice of VS Code's `EnvironmentVariableCollection` that putting `gemdb` + * on the PATH needs, so the decision is testable without an editor. + */ +export interface TerminalEnvironment { + clear(): void; + prepend(variable: string, value: string): void; +} + +/** + * Put `gemdb` on the PATH of terminals VS Code opens. + * + * The command is generated into `/bin`, which is on nobody's PATH, + * so `which gemdb` in a VS Code terminal reported nothing and the README's + * answer was an `export PATH=...` line the user had to add to their own + * profile. VS Code contributes this itself, per extension: it applies to + * terminals this editor launches, is reverted when GemDB is disabled or + * uninstalled, and never touches the user's shell profile — which keeps it on + * the automated side of the line (inert, reversible, and not a change to the + * machine), where editing a profile would not be. + * + * `clear()` first because the collection is persisted across window reloads: + * re-applying on activation is what keeps it right after the root path + * changes, and without the clear it would accumulate a stale entry per + * reload. Contributed whether or not `bin/gemdb` exists yet — on first run it + * does not, and a terminal opened during setup should find the command once + * setup writes it rather than needing to be reopened. + */ +export function putCliOnPath(env: TerminalEnvironment): void { + env.clear(); + env.prepend('PATH', `${cliDirPath()}${path.delimiter}`); +} + /** * Guarantee `/bin/gemdb` exists and matches this build, for callers * about to hand that path to something else. @@ -165,6 +198,17 @@ export function writeCliScripts(extensionPath: string): void { // Grail's Python exceptions live outside the Error branch, which is why // grail.tpz's own file mode exits 0 on a Python error. // + // There is deliberately no `exit` command at the end. topaz's own help says + // it of `-S`: the script is processed with INPUT, "topaz exits when the + // script completes", and "exit and quit commands are ignored". Ignored + // quietly when stdin is a pipe, which is why both suites and every run + // through a non-interactive shell missed it — but out loud when stdin is a + // tty, where `gemdb hide.py` printed four lines of topaz explaining that it + // was ignoring the EXIT, plus a "Logging out session 1." the ignored exit + // provoked. The status still leaves through GEMDB_STATUS_FILE, which never + // depended on that line (measured again without it: sys.exit(3) → 3, an + // uncaught exception → 1, a missing file → 2). + // // sys.exit(n) is decoded here, not upstream: Grail raises its own SystemExit // (never ExitClientError — input()'s except SystemExit and finally blocks // must keep working), and the exit argument survives only in the exception's @@ -196,8 +240,19 @@ warnings) consult SessionTemps #GrailConsole first; that write is transient and leaves System needsCommit untouched. Boxed in an Array because that is the #GrailConsole protocol (see builtins.gs ___console___: SessionTemps at:put: sends to what it stores, which a -ClientForwarder cannot survive; a GsFile could, but one protocol)." -SessionTemps current at: #'GrailConsole' put: (Array with: GsFile stdout). +ClientForwarder cannot survive; a GsFile could, but one protocol). + +The second slot declares what this sink takes: a GsFile takes BYTES, so +every console write must be encoded. Without it Grail wrote a Unicode16's +code units straight to stdout -- a NUL between every ASCII character, and +a non-ASCII character truncated to its low byte -- so print('caf\u00e9') +was UTF-16BE on the terminal while the same print through the shell (a +character stream over GCI) was right. Declared here rather than sniffed +there because the sink cannot be asked: the shell's sink is a +ClientForwarder, and asking one anything -- class, respondsTo:, isNil -- +forwards to the client as error 2336, which is not catchable in the gem." +SessionTemps current at: #'GrailConsole' + put: (Array with: GsFile stdout with: #'utf8'). [ [ target := args at: ofs + 1. @@ -259,7 +314,6 @@ SessionTemps current at: #'GrailConsole' put: (Array with: GsFile stdout). f nextPutAll: status printString; close]. ]. % -exit 0 `; const script = `#!/bin/bash diff --git a/src/extension.ts b/src/extension.ts index c9bb98f..bd92dad 100644 --- a/src/extension.ts +++ b/src/extension.ts @@ -12,6 +12,7 @@ import { uninstall, } from './lifecycle'; import { autoStartSuppressed, initAutoStart, suppressAutoStart } from './autoStart'; +import { cliDirPath, putCliOnPath } from './cli'; import { withSetupLock } from './lock'; import { disposeLog, log, showLog } from './log'; import { @@ -176,6 +177,8 @@ export function activate(context: vscode.ExtensionContext): void { ) { // Every notebook's session is bound to the old database too. logoutAll(); + // And the `gemdb` on the PATH of new terminals is the old root's. + if (isSupportedPlatform()) putCliOnPath(context.environmentVariableCollection); status.refresh(); } }), @@ -190,6 +193,15 @@ export function activate(context: vscode.ExtensionContext): void { return; } + // `which gemdb` should answer in a terminal opened here. After the platform + // gate: on a platform GemDB cannot run, there will never be a command to + // find, and pointing the PATH at a directory that stays empty explains + // nothing. + // Named with the real directory, not `~/GemDB/bin`: the root path is a + // setting, and this string is what the terminal shows to explain itself. + context.environmentVariableCollection.description = `Adds the \`gemdb\` command (${cliDirPath()}) to the PATH.`; + putCliOnPath(context.environmentVariableCollection); + status.refresh(); void prepareOnFirstRun(context, extensionPath, () => status.refresh()).then(() => diff --git a/src/notebook.ts b/src/notebook.ts index 1cb4ecb..5c22cdb 100644 --- a/src/notebook.ts +++ b/src/notebook.ts @@ -192,19 +192,22 @@ export async function resetActiveNotebook(): Promise { /** * Open a new notebook with one Python cell, ready to run. * - * The starter cell is not decoration: `import gemstone` is the one thing that + * The starter cell is not decoration: `import gemdb` is the one thing that * makes this different from any other Python notebook, and showing it here is - * cheaper than explaining it. + * cheaper than explaining it. `gemdb` is the module a developer should meet + * first — `gemstone` is Grail's own lower-level surface, and a starter cell + * that opens with it teaches the wrong name. */ export async function newNotebook(): Promise { const starter = [ '# Python here runs inside your GemDB database.', - '# The gemstone module reaches the data stored in it.', - 'import gemstone', + '# Everything reachable from gemdb.root is still there tomorrow.', + 'import gemdb', '', - 'gemstone["greeting"] = "Hello from GemDB!"', - 'gemstone.system.commit()', - 'gemstone["greeting"]', + 'gemdb.root["greeting"] = "Hello from GemDB!"', + 'gemdb.commit()', + '', + 'gemdb.root["greeting"]', ].join('\n'); const cell = new vscode.NotebookCellData(vscode.NotebookCellKind.Code, starter, 'python');