diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1305c115..57f1756d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -128,3 +128,9 @@ jobs: - name: go test run: go test ./... + + # Informational, never fails: the tests in internal/i18n are the gate. This prints + # per-language coverage so a translation PR shows up in the log with its progress, + # and so a language falling under the floor is visible before a user meets it. + - name: Translation coverage + run: go run ./cmd/i18n-coverage diff --git a/CHANGELOG.md b/CHANGELOG.md index ac6426f4..45791e8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,23 @@ exact tag (`ghcr.io/calnode/calnode:0.1.0`) if you need stability between upgrad ## [Unreleased] +### Changed + +- **Partial locale catalogues are allowed, so a language can be translated in stages.** A + locale file no longer has to contain every English key before it can merge: a missing key + falls back to English at lookup time (`Locale.T`). This was previously blocked by + `TestAllLocalesHaveTheSameKeys`, which asserted exact set equality in both directions, so a + half-finished locale was unmergeable and a volunteer had to produce all 169 strings before + contributing anything. That is the practical reason community translations stall. + A locale is now offered to visitors - language switcher, `Accept-Language` matching and the + public API payload - only at or above `MinCoverage` (80% of the English keys), so nobody is + served a mostly-English booking page by accident. Below the floor `?lang=` still + resolves, so whoever is translating it can check their work, and startup logs which locales + are withheld and why. The unknown-key check survives as `TestLocalesHaveNoUnknownKeys`, + since a key English does not have is always a typo, and the floor is pinned by + `TestOfferedLocalesMeetCoverageFloor`. No shipped locale changes: all 11 are complete and + remain offered. + ## [0.13.0] - 2026-11-03 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cb3fcad8..86538c89 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -74,9 +74,9 @@ prefer additive, nullable/defaulted columns. ## Translations -Calnode ships 11 locales (`en es fr fr-CA de el it nl pl pt sv`) across the booker-facing surfaces: +Calnode ships 11 locales (`en es fr fr-CA de el it nl pl pt sv`) on the booker-facing surfaces: booking page, manage/reschedule page, embed widget, the four emails, and the calendar -invite. The admin UI and the built-in video room are English-only. +invite. The admin SPA has its own catalogue, in progress. **Translation PRs are very welcome** - both new languages and corrections to existing ones. **Most non-English locales are machine drafts with no native review, and none has @@ -84,24 +84,39 @@ had a full native read** (Polish was contributed and grammar-checked by a native speaker), so if you speak one of them, fixes are genuinely valuable and will be merged readily. Small, single-language PRs are easier to review than sweeping ones. -**Adding a language is adding one file:** `internal/i18n/locales/.json`. No Go, -template, or frontend change is needed - `init()` globs the directory, and the language -switcher, the fallback-language setting and the public API payload all read -`SupportedLocales()`. - -1. Copy `internal/i18n/locales/en.json` and translate the values. Keep every key, keep - the key order (it keeps diffs readable), and keep printf verbs (`%s`, `%d`, `%q`) - intact - reordering them is fine with indexed verbs (`%[2]s`), dropping them is not. -2. Name the file **BCP-47 canonical**: `pt-BR.json`, never `pt-br.json`. -3. `dow_short_*`, `month_short_*`, `date_format` and `clock_format` are data, not code - +A language is **one file, and it does not have to be complete.** + +1. Copy `internal/i18n/locales/en.json` to `internal/i18n/locales/.json` and translate + the values. Delete the keys you have not got to yet if that helps; a missing key falls + back to English at lookup time, so a partial file is mergeable and useful. +2. Keep the keys spelled exactly as English has them. A key that is not in `en.json` fails + the build (`TestLocalesHaveNoUnknownKeys`), because it is always a typo and renders for + nobody. +3. Keep printf verbs intact: `%s`, `%d`, `%q`. Reordering is fine with indexed verbs + (`%[2]s`); dropping one is not. In the admin catalogue the placeholders are `{name}`. +4. Name the file **BCP-47 canonical**: `pt-BR.json`, never `pt-br.json`. +5. `dow_short_*`, `month_short_*`, `date_format` and `clock_format` are data, not code - Go has no locale date tables. Match CLDR; the test below checks you against `Intl`. -4. Run `go test ./internal/i18n/`. Three guards will tell you exactly what is wrong: - key parity, printf-verb parity (with `fmt` as the oracle), and a CLDR cross-check of - the date tables (needs `node` on PATH; skipped if absent). -5. Eyeball the result: `go run ./cmd/calnode`, then +6. Run `go test ./internal/i18n/` and `make i18n-coverage` to see where you stand. +7. Eyeball the result: `go run ./cmd/calnode`, then `http://localhost:3000/book/?lang=` and the manage page. Long words overflowing buttons is the usual surprise; check mobile width too. +**When a language is offered.** A language appears in the switcher, and is matched from +`Accept-Language`, only at or above 80% of the English keys (`i18n.MinCoverage`). Below the +floor it still resolves with `?lang=`, so you can check your work in place, and +startup logs which languages are withheld. The floor exists because a mostly-English +booking page is worse for a booker than English is. + +**What passing tests does not prove.** They check structure, not wording. Before a language +is advertised, a speaker should read the booking page, the manage page and the four emails +end to end. The usual misses are literal calques, agreement when two translated sentences +are spliced together, and plurals written as flat strings. + +Terminology is agreed in [`docs/i18n-glossary.md`](docs/i18n-glossary.md). Add your +language's column there rather than inventing a second word for something that already has +one. + Background and the full list of limitations (2-form plurals, no RTL) are in [ARCHITECTURE.md §23](docs/ARCHITECTURE.md). diff --git a/Makefile b/Makefile index b2975719..63930971 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: frontend backend build +.PHONY: frontend backend build i18n-coverage frontend: cd frontend && pnpm install && pnpm build @@ -7,3 +7,8 @@ backend: frontend go build -o calnode ./cmd/calnode build: backend + +# Per-language translation coverage, for both catalogues. Informational; the tests in +# internal/i18n are the gate. +i18n-coverage: + go run ./cmd/i18n-coverage diff --git a/cmd/i18n-coverage/main.go b/cmd/i18n-coverage/main.go new file mode 100644 index 00000000..0f13662c --- /dev/null +++ b/cmd/i18n-coverage/main.go @@ -0,0 +1,157 @@ +// Command i18n-coverage prints how much of each catalogue every shipped language +// carries. +// +// It is a progress report, not a gate: the tests in internal/i18n are what fail a +// build. This exists because a translator needs to see 512/1324 rather than guess, and +// a reviewer needs to see whether a language is close enough to offer before reading it. +// +// Run it from the repository root: +// +// go run ./cmd/i18n-coverage +// make i18n-coverage +package main + +import ( + "encoding/json" + "flag" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + + "github.com/calnode/calnode/internal/i18n" +) + +type catalogue struct { + name string + dir string + // floor is the coverage below which the language is not offered to visitors. + // Zero means "no floor for this catalogue yet". + floor float64 +} + +const sourceCode = "en" + +func main() { + root := flag.String("root", ".", "repository root") + flag.Parse() + + catalogues := []catalogue{ + {name: "booker (public surfaces)", dir: "internal/i18n/locales", floor: i18n.MinCoverage}, + {name: "admin (SPA)", dir: "frontend/src/lib/i18n", floor: 0}, + } + + for _, c := range catalogues { + if err := report(*root, c); err != nil { + fmt.Fprintf(os.Stderr, "i18n-coverage: %s: %v\n", c.dir, err) + os.Exit(1) + } + } +} + +func report(root string, c catalogue) error { + dir := filepath.Join(root, c.dir) + entries, err := os.ReadDir(dir) + if os.IsNotExist(err) { + fmt.Printf("%s\n %s does not exist yet, skipped\n\n", c.name, c.dir) + return nil + } + if err != nil { + return err + } + + source, err := load(filepath.Join(dir, sourceCode+".json")) + if err != nil { + return err + } + if len(source) == 0 { + return fmt.Errorf("%s has no strings", filepath.Join(dir, sourceCode+".json")) + } + + type row struct { + code string + keys int + cover float64 + unknown int + } + var rows []row + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".json") { + continue + } + code := strings.TrimSuffix(e.Name(), ".json") + strings_, err := load(filepath.Join(dir, e.Name())) + if err != nil { + return err + } + r := row{code: code, keys: len(strings_)} + for k := range strings_ { + if _, ok := source[k]; ok { + r.cover++ + } else { + r.unknown++ + } + } + r.cover = float64(r.cover) / float64(len(source)) + rows = append(rows, r) + } + sort.Slice(rows, func(i, j int) bool { return rows[i].code < rows[j].code }) + + fmt.Printf("%s\n %s (source %s.json, %d keys)\n\n", c.name, c.dir, sourceCode, len(source)) + fmt.Printf(" %-7s %6s %8s %10s %8s\n", "code", "keys", "coverage", "status", "unknown") + for _, r := range rows { + status := "source" + switch { + case r.code == sourceCode: + case r.unknown > 0: + status = "has unknown keys" + case c.floor > 0 && r.cover < c.floor: + status = "withheld" + default: + status = "offered" + } + fmt.Printf(" %-7s %6d %7.0f%% %10s %8d\n", r.code, r.keys, r.cover*100, status, r.unknown) + } + if c.floor > 0 { + fmt.Printf("\n floor: %.0f%% (below it a language is not offered; ?lang= still works)\n", c.floor*100) + } + fmt.Println() + return nil +} + +// load reads one catalogue file and returns its flattened key set. Nested objects are +// joined with dots, so the admin catalogue's plural members count individually +// ("common.save", "teams.memberCount.one", ...). Arrays are not used in either +// catalogue and are skipped rather than guessed at. +func load(path string) (map[string]struct{}, error) { + // #nosec G304 -- path is the repository root this developer tool was invoked from + // plus a fixed catalogue name; there is no untrusted input, and this binary is never + // served. It reports on files a contributor is already editing. + b, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var raw map[string]any + if err := json.Unmarshal(b, &raw); err != nil { + return nil, fmt.Errorf("%s: %w", path, err) + } + out := map[string]struct{}{} + flatten("", raw, out) + return out, nil +} + +func flatten(prefix string, v any, out map[string]struct{}) { + switch t := v.(type) { + case map[string]any: + for k, child := range t { + key := k + if prefix != "" { + key = prefix + "." + k + } + flatten(key, child, out) + } + case string: + out[prefix] = struct{}{} + } +} diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 73d63b6a..426cbb39 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1308,9 +1308,28 @@ you is a key in the JSON instead (`internal/i18n/datetime.go`): `date_format`, ` `dow_short_*`, `month_short_*`. That also makes per-language date shape expressible - German carries its own `date_format` for the ordinal period ("Mo, 22. Jun 2026"). +A locale does **not** have to be complete to merge. Partial catalogues are supported: any +key a locale lacks falls back to English at lookup time (`Locale.T`), so a translation can +land in stages instead of in one block. The practical reason is that nobody translates 169 +strings in one sitting, and requiring completeness made an in-progress language +unmergeable - which is how community translations quietly die. + +The other half of that bargain is `MinCoverage` (80% of the English keys). A locale at or +above it is offered to visitors: the language switcher, `Accept-Language` matching and the +public API payload all read `SupportedLocales()`, which filters on it. A locale below the +floor still exists and `?lang=` still resolves, so whoever is translating it can +check their work in situ, but no visitor is served a mostly-English booking page by +accident. The floor is set with headroom because the English catalogue grows: adding +strings can push a locale under it, and that is the signal to chase the translators. +Startup logs a line for every locale it withholds and why. + Three guards police a new file (`go test ./internal/i18n/`): -- `TestAllLocalesHaveTheSameKeys` - no missing or unknown keys. +- `TestLocalesHaveNoUnknownKeys` - no key that English does not have. A missing key is + fine now (that is the fallback), but an unknown one is always a typo or a renamed key + and would render for nobody. This replaced `TestAllLocalesHaveTheSameKeys`, which + asserted set equality in both directions and therefore rejected the in-progress locales + the floor exists to support. - `TestAllLocalesHaveMatchingFormatVerbs` - uses `fmt` **itself** as the oracle, so a locale that drops or reorders a `%s`/`%d` fails the test instead of emitting `%!(EXTRA…)` to a booker. An earlier hand-rolled parser for this was unsound (it missed indexed verbs like @@ -1320,6 +1339,9 @@ Three guards police a new file (`go test ./internal/i18n/`): only check that can tell `dim.` from `dim`, or catch a 12h/24h call that is wrong for the language. It caught a real drift on first run (`es` had `sep` where CLDR says `sept`). +`TestOfferedLocalesMeetCoverageFloor` pins the invariant the three guards do not: nothing +below `MinCoverage` is ever in `supported`. + Two traps: - **Filenames must be BCP-47 canonical.** `pt-br.json` is wrong - `language.Make("pt-br")` diff --git a/docs/i18n-glossary.md b/docs/i18n-glossary.md new file mode 100644 index 00000000..2ee884b5 --- /dev/null +++ b/docs/i18n-glossary.md @@ -0,0 +1,64 @@ +# Translation glossary + +One concept, one word, per language. The point is that a booking does not become a +"rendez-vous" on one screen and a "réservation" on the next, and that a new translator does +not have to invent terminology that is already decided. + +**How to use it** + +- Before translating a term, look it up here. If it is missing, add it with its key. +- The values below were read from the shipped catalogues, not invented. If a catalogue + disagrees with this table, the table is wrong and should be fixed in the same PR. +- **Adding a language means adding a column.** Do not introduce a second word for a concept + that already has one. +- Two catalogues exist: the **booker** surfaces (`internal/i18n/locales`) and the **admin** + SPA (`frontend/src/lib/i18n`). They are separate systems with separate registers, and they + currently disagree in places. Those rows are marked **divergent** and are decisions, not + accidents. + +## Core terms + +| Concept | English | French (booker) | French (admin) | Notes | +|---|---|---|---|---| +| booking (the object) | booking | réservation | réservation | Never "rendez-vous" for the object. | +| to book (verb) | book | réserver | réserver | | +| event type | event type | - | type de rendez-vous | **divergent**: the booker has no public name for it. | +| host | host | - | hôte | The booker addresses the booker, not the host. | +| attendee / booker | attendee | - | participant | Use "participant" in admin; the booker is "vous". | +| team | team | équipe | équipe | | +| team member | team member | membre de l'équipe | membre | | +| availability | availability | disponibilités | disponibilités | Plural in French. | +| slot / time slot | slot | créneau | créneau | | +| timezone | timezone | fuseau horaire | fuseau horaire | | +| reschedule | reschedule | reprogrammer | reprogrammer | Noun: "reprogrammation". | +| cancel | cancel | annuler | annuler | Noun: "annulation". | +| round robin | round robin | - | tourniquet | French admin uses "tourniquet"; do not calque "tourner entre eux". | +| reminder | reminder | rappel | rappel | | +| directory | directory | - | annuaire | Only the person and team pages. | + +## Register + +| Surface | Register | Reason | +|---|---|---| +| Booker | Formal, second person plural (`vous`) | A customer booking a call from a business page. | +| Admin | Formal (`vous`), terse | An operator working in the tool all day. | + +The existing booker locales are deliberately split by market: `fr`/`de`/`it`/`pt` formal, +`es`/`nl`/`sv` informal. Match the language, not the English. + +## Things that are not translated + +Brand names, URLs, protocol tokens (`STARTTLS`, `RSVP`, `PII`), third-party console labels +("APIs & Services", "Authorised redirect URIs"), and code identifiers. Translating those is +almost always a mistake; leave them byte-identical. + +## Placeholders + +Never translate a placeholder. `%s`, `%d`, `%q` (booker) and `{name}` (admin) must survive +verbatim, and a missing one is a runtime bug rather than a stylistic choice. The tests check +this, but a test cannot see that you moved a word into the wrong place. + +## Cross-references + +- `CONTRIBUTING.md` - how to add or extend a language +- `docs/ARCHITECTURE.md` §23 - how the catalogue system works and its limitations diff --git a/internal/i18n/i18n.go b/internal/i18n/i18n.go index 44ad0eaa..8c7270a0 100644 --- a/internal/i18n/i18n.go +++ b/internal/i18n/i18n.go @@ -11,6 +11,7 @@ import ( "embed" "encoding/json" "fmt" + "log" "sort" "golang.org/x/text/language" @@ -23,6 +24,23 @@ var localeFS embed.FS // DefaultCode is the fallback locale when nothing else matches. const DefaultCode = "en" +// MinCoverage is the fraction of English keys a locale must carry before it is offered +// to visitors (language switcher, Accept-Language matching, public API payload). +// +// Locales may now be translated incrementally: a locale file does not have to be +// complete to merge, and any key it lacks falls back to English at lookup time (see +// Locale.T). That is what makes an outside contribution worth landing at all - nobody +// completes 169 strings in one sitting, and a half-finished locale used to be +// unmergeable. The floor is the other half of that bargain: a mostly-English "German" +// booking page is worse for a booker than English, so a locale below this is kept out +// of the picker entirely rather than half-offered. Get() still returns it, so an +// explicit ?lang= works for whoever is translating it. +// +// Set with headroom, because the English catalogue grows: at 0.8 a locale survives +// roughly a fifth more English keys before it drops out, which is the signal to chase +// the translators rather than a reason to panic. +const MinCoverage = 0.8 + // Locale holds one language's resolved string table. type Locale struct { Code string @@ -81,12 +99,34 @@ func init() { supportedCodes = append(supportedCodes, DefaultCode) supported = append(supported, language.Make(DefaultCode)) for _, code := range otherCodes { + // A partial catalogue is allowed to exist and to be merged (see MinCoverage), + // but it is not offered to visitors until it is substantially translated. + if !offeredToVisitors(locales[code]) { + log.Printf("i18n: locale %q is %.0f%% translated (floor %.0f%%); not offered to visitors - ?lang=%s still works", + code, Coverage(locales[code])*100, MinCoverage*100, code) + continue + } supportedCodes = append(supportedCodes, code) supported = append(supported, language.Make(code)) } matcher = language.NewMatcher(supported) } +// Coverage reports how much of the English catalogue a locale carries, as a fraction. +// English is 1 by definition; a locale with keys English does not have can exceed 1, +// which TestLocalesHaveNoUnknownKeys prevents in practice. +func Coverage(l *Locale) float64 { + en := Default() + if l == nil || en == nil || len(en.strings) == 0 { + return 0 + } + return float64(len(l.strings)) / float64(len(en.strings)) +} + +// offeredToVisitors is the single place the floor is applied, so init and the tests +// cannot disagree about what "offered" means. +func offeredToVisitors(l *Locale) bool { return Coverage(l) >= MinCoverage } + // Default returns the fallback (English) locale. func Default() *Locale { return locales[DefaultCode] } diff --git a/internal/i18n/i18n_test.go b/internal/i18n/i18n_test.go index e3afc581..7ad6423b 100644 --- a/internal/i18n/i18n_test.go +++ b/internal/i18n/i18n_test.go @@ -93,8 +93,19 @@ func TestJSON_roundTrips(t *testing.T) { func TestSupportedLocales(t *testing.T) { opts := SupportedLocales() - if len(opts) != len(locales) { - t.Fatalf("SupportedLocales() returned %d entries, want %d", len(opts), len(locales)) + // Every locale at or above the coverage floor is offered, and nothing below it is. + // This used to assert len(opts) == len(locales), which is only true while every + // shipped locale is complete; with partial catalogues allowed that invariant is + // wrong, and it would fail the first time a translation landed in stages. + want := 0 + for _, l := range locales { + if offeredToVisitors(l) { + want++ + } + } + if len(opts) != want { + t.Fatalf("SupportedLocales() returned %d entries, want %d (locales at or above the %.0f%% floor)", + len(opts), want, MinCoverage*100) } if opts[0].Code != DefaultCode { t.Errorf("SupportedLocales()[0] = %q, want English (%q) first", opts[0].Code, DefaultCode) @@ -258,17 +269,21 @@ func TestFormatDateTime(t *testing.T) { } } -func TestAllLocalesHaveTheSameKeys(t *testing.T) { +// TestLocalesHaveNoUnknownKeys is the half of the old completeness gate that still +// matters. A locale no longer has to be complete to merge: partial catalogues are +// allowed, and a key a locale lacks falls back to English at lookup time (Locale.T), so +// a translator can land a language in stages instead of in one unmergeable 169-string +// block. What remains an error is a key English does not have - that is always a typo or +// a renamed key, it renders for nobody, and it would otherwise rot silently. +// +// The old test asserted set equality in both directions. Dropping the "missing key" half +// is the point of this change; see MinCoverage for the guard that replaces it. +func TestLocalesHaveNoUnknownKeys(t *testing.T) { en := Default() for code, l := range locales { if code == DefaultCode { continue } - for k := range en.strings { - if _, ok := l.strings[k]; !ok { - t.Errorf("locale %q is missing key %q (present in English)", code, k) - } - } for k := range l.strings { if _, ok := en.strings[k]; !ok { t.Errorf("locale %q has key %q that doesn't exist in English", code, k) @@ -277,6 +292,56 @@ func TestAllLocalesHaveTheSameKeys(t *testing.T) { } } +// TestOfferedLocalesMeetCoverageFloor pins the invariant the language switcher, the +// Accept-Language matcher and the public API payload all rely on: everything in +// `supported` is at or above MinCoverage. A locale below the floor must be reachable +// only by an explicit code (?lang=), never offered to a visitor who did not ask for it. +func TestOfferedLocalesMeetCoverageFloor(t *testing.T) { + if len(supportedCodes) == 0 { + t.Fatal("no supported locales") + } + for _, code := range supportedCodes { + cov := Coverage(locales[code]) + if cov < MinCoverage { + t.Errorf("offered locale %q is %.0f%% translated, below the %.0f%% floor", + code, cov*100, MinCoverage*100) + } + } + if supportedCodes[0] != DefaultCode { + t.Errorf("English must stay first in supported (matcher fallback), got %q", supportedCodes[0]) + } +} + +// TestCoverageFloorHidesPartialLocale exercises the decision itself with synthetic +// locales, so it does not depend on shipping a half-translated file to prove the rule. +func TestCoverageFloorHidesPartialLocale(t *testing.T) { + en := Default() + total := len(en.strings) + + partial := &Locale{Code: "xx", strings: make(map[string]string, total)} + for k := range en.strings { + if len(partial.strings) >= int(float64(total)*MinCoverage)-1 { + break + } + partial.strings[k] = "translated" + } + if offeredToVisitors(partial) { + t.Errorf("a locale just below the floor (%d/%d keys) must not be offered", len(partial.strings), total) + } + + full := &Locale{Code: "xy", strings: make(map[string]string, total)} + for k, v := range en.strings { + full.strings[k] = v + } + if !offeredToVisitors(full) { + t.Error("a complete locale must be offered") + } + + if got := Coverage(nil); got != 0 { + t.Errorf("Coverage(nil) = %v, want 0", got) + } +} + // enFormatArgs derives probe arguments for an English format string by walking its // printf verbs. It only ever parses ENGLISH — the strings we author and control, which // use plain %s/%d/%q or clean %[n]-indexed forms. Translations are never parsed; they're