Normalize non-standard language codes to ISO 639-1 at XLS ingestion - #23
Merged
Conversation
IEC CDD source .xls files use "jp" for Japanese instead of the ISO 639-1 code "ja". The gem previously passed these through verbatim, creating a mismatch with the browser's CSS visibility rules and the LanguageSwitcher, which expect ISO codes. Add Languages.normalize(code) — a class method that maps known non-conformant codes to their ISO equivalents and emits a warning on stderr so data-quality issues are visible. The LANG_ALIASES map is extensible for future non-standard codes. Apply normalization at the two points where language codes enter from the XLS source: 1. SheetSchema.canonical_id — column IDs like MDC_P004.jp are now canonicalized to MDC_P004.ja before entering the entity model. 2. SheetSchema#lang_hash_for — directive-row language keys (PROPERTY_NAME.jp etc.) are normalized before building the per-column name/definition/note language hashes. Also apply normalization in Languages#initialize and Languages.from_properties so any code path that constructs a Languages object gets clean codes. The warning fires once per non-conformant code encountered, making it easy to audit which dictionaries still carry legacy codes.
7 tasks
The original PR landed language normalization inside Opencdd::Languages
(the model class), where it had three architectural problems:
- emitted warn on stderr per call (log spam in batch ingestion),
- coupled a model object to a source-format concern,
- and applied the same rewrite at five different sites.
This commit applies Option A from the audit:
- Opencdd::Parcel::LanguageAliases owns the jp→ja alias table as a
Parcel-format concern. The aliases are an artefact of one specific
source format (IEC CDD .xls exports), not of the canonical CDD
model.
- SheetSchema applies normalization at the ingestion boundary
(canonical_id + lang_hash_for); the per-call stderr warn is
removed, eliminating log spam in batch builds.
- SheetSchema exposes #normalized_language_codes (frozen Hash
{original => normalized}) for structured data-quality reporting
-- replaces the noisy warn with a per-schema audit trail.
- Opencdd::Languages is now a pure value object: it assumes
ISO 639-1 codes and does not rewrite at construction or query
time. Source-format quirks stay at the format layer.
- Drop redundant nil/empty guard in SheetSchema#lang_hash_for.
- Use `_` for the unused prefix destructure in
Languages.from_properties.
Specs: new spec/parcel/language_aliases_spec covers the pure mapping;
spec/parcel/sheet_schema_spec adds #normalized_language_codes coverage.
spec/languages_spec drops .normalize blocks and adds a "model contract"
describe documenting that Languages does not rewrite.
Closes #23.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
IEC CDD source
.xlsfiles usejpfor Japanese instead of the ISO639-1 code
ja. The gem previously passed these through verbatim,creating a mismatch with the browser's CSS visibility rules and the
LanguageSwitcher, which expect ISO codes.
Fix
Language normalization is now applied at the SheetSchema ingestion
boundary only. Source-format quirks live in a dedicated module
where they belong — they are an artefact of one specific source
format, not of the canonical CDD model.
Opencdd::Parcel::LanguageAliasesNew module owning the alias table. Pure mapping (no I/O, no side
effects):
Extensible — add new aliases as needed.
Opencdd::Parcel::SheetSchemaApplies normalization at exactly two sites:
canonical_id— column IDs likeMDC_P004.jp→MDC_P004.ja.lang_hash_for— directive-row language keys(
PROPERTY_NAME.jp) →ja.Exposes
#normalized_language_codes(frozenHash{ original => normalized })as a structured, per-schema audit trail — replaces the previous
per-call stderr warn, which would have produced thousands of repeated
lines in batch ingestion (e.g.
iec61987has 11,831 entities).Opencdd::LanguagesReverted to a pure value object. Assumes ISO 639-1 codes
unconditionally and does not rewrite at construction or query time.
The model contract is documented in
spec/languages_spec.rb.Test plan
bundle exec rspec— 952 examples, 0 failuresspec/parcel/language_aliases_spec.rb— new spec covering thepure mapping: jp→ja, ISO passthrough, nil/empty, whitespace,
no side effects, idempotence, frozen ALIASES, valid ISO values.
spec/languages_spec.rb— model contract specs verify thatLanguages.new(source: "jp")does NOT rewrite, andinclude?does NOT rewrite.
spec/parcel/sheet_schema_spec.rb—#normalized_language_codescovers: empty for clean sources, populates from directive keys
and PROPERTY_ID values, frozen, dedupes repeated codes.