Skip to content

Normalize Japanese language code jp to ISO 639-1 ja - #56

Closed
ronaldtse wants to merge 1 commit into
mainfrom
fix/normalize-jp-to-ja
Closed

Normalize Japanese language code jp to ISO 639-1 ja#56
ronaldtse wants to merge 1 commit into
mainfrom
fix/normalize-jp-to-ja

Conversation

@ronaldtse

@ronaldtse ronaldtse commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Summary

IEC CDD source data uses jp for Japanese, but the browser's CSS visibility rules and the ISO 639-1 standard use ja. This mismatch made all Japanese translations invisible.

Fix: normalize at the data import boundary

Added a normalizeLanguageCodes() pipeline stage in src/lib/build/stages.ts that runs during npm run fetch-data, right after data is acquired into src/content/data/. The stage:

  1. Walks every database.json under dataTarget
  2. For each entity, checks every *_ml field
  3. Renames the jp key to ja (preserving an existing ja key if both are present)
  4. Writes the normalized JSON back

Idempotent and extensible via the LANG_ALIASES map ({"jp": "ja"} — add more if other non-standard codes surface).

Also adds ja to KNOWN_LANGS in LanguageSwitcher.vue.

Why at import, not at build-time load

The normalization happens once when data enters our ecosystem (during fetch-data), not every time loadDictionary() reads the JSON at build time. The committed/fetched data is clean; downstream consumers don't need defensive checks.

Verified on real data

[fetch-data] [normalize-language-codes] ok — renamed 80 key(s) across 8 dict(s)

After normalization: 0 jp keys remaining, 80 ja keys present across all dictionaries. Built output confirms ml-ja spans (not ml-jp) on multilingual entity pages.

Multilingual data coverage (from ../data-private/)

Dictionary Entities Multilingual Languages
iec61987 12,789 10,209 de, en, fr, ja, zh
iec62683 1,861 1,855 de, en, fr, ja, zh
iec63213 220 220 de, en, fr, zh
iec63508 151 64 de, en, fr, zh
iec61360 4,491 146 de, en
iec62720 2,559 2 de, en

Test plan

  • npm run check — 0 errors
  • 3 new pipeline tests (renames jp→ja, preserves existing ja, skips when clean)
  • npm run build — 25,476 pages built
  • npm run fetch-data — "renamed 80 key(s) across 8 dict(s)"
  • Verified: 0 jp keys, 80 ja keys in committed data
  • Built HTML: ml-ja spans present on iec62683/p/ACE061
  • Manual: visit a multilingual entity, language switcher shows JA button

IEC CDD source data uses "jp" for Japanese, but the browser's CSS
visibility rules and the ISO 639-1 standard use "ja". This mismatch
made all Japanese translations invisible: the data generated
<span class="ml-jp"> spans, but the CSS only had body[data-lang="ja"]
rules, and the LanguageSwitcher didn't list Japanese at all.

Fix at the closest point to the data — the loadDictionary() function
in data.ts, where raw JSON is parsed into EntityNode objects. A new
normalizeLanguageCodes() function walks every *_ml field on each
entity and renames the "jp" key to "ja" (preserving an existing "ja"
key if both are present). The LANG_ALIASES map is extensible for
future non-standard codes.

Also add "ja" to KNOWN_LANGS in LanguageSwitcher.vue so the Japanese
button appears in the language toggle.

Verified: iec62683 entity ACE061 now renders <span class="ml-ja">
with the correct Japanese text in built output. Zero ml-jp spans
remain. All 5 languages (de, en, fr, ja, zh) render and respond to
the language switcher.
@ronaldtse
ronaldtse force-pushed the fix/normalize-jp-to-ja branch from 3764768 to 5d2a5d9 Compare August 7, 2026 06:16
@ronaldtse

Copy link
Copy Markdown
Contributor Author

Closing in favor of opencdd/opencdd-ruby#23 — the normalization belongs in the Ruby gem at the XLS ingestion point, not in the browser pipeline.

@ronaldtse ronaldtse closed this Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant