From 981a3b6d7eeef2a8ad1335b62fc79b14f05c4f91 Mon Sep 17 00:00:00 2001 From: Julien Dehais Date: Thu, 15 Oct 2026 08:13:54 +0200 Subject: [PATCH 1/5] feat: keep the parsed translation files in a cache pool keyed by path, date and size A view over every catalogue parsed every translation file again in each new process. TranslationFileScanner now keeps each parsed file in the pool cyllene_digital_ai_translation.translation_files_cache (adapter cache.app, 7 days, both overridable under framework.cache.pools), keyed by its path, date of modification, size and a format version. A file that changes gets a new key; the walk of the directories is never cached. An unparseable file stays reported when served from the cache. On by default, translation_files_cache: false turns it off. The scanner implements ResetInterface, so a Messenger worker sees an edited file between two messages. ADR 0027. --- CLAUDE.md | 8 ++ config/definition.php | 4 + ...0027-parsed-files-cached-by-fingerprint.md | 42 +++++++ docs/adr/README.md | 1 + docs/configuration-reference.md | 23 ++++ docs/operations/cache.md | 26 ++++- docs/operations/multi-server.md | 5 + docs/public-contract.md | 2 + src/Catalogue/TranslationFileScanner.php | 72 ++++++++++-- src/CylleneDigitalAiTranslationBundle.php | 27 ++++- tests/App/FullConfigTestKernel.php | 1 + .../TranslationFilesCacheWiringTest.php | 66 +++++++++++ .../TranslationFileScannerCacheTest.php | 108 ++++++++++++++++++ 13 files changed, 376 insertions(+), 9 deletions(-) create mode 100644 docs/adr/0027-parsed-files-cached-by-fingerprint.md create mode 100644 tests/Functional/TranslationFilesCacheWiringTest.php create mode 100644 tests/Unit/Catalogue/TranslationFileScannerCacheTest.php diff --git a/CLAUDE.md b/CLAUDE.md index b6e9326..30de534 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -155,6 +155,14 @@ facade - is in the records 0016, 0017, 0022 and 0023: read them before "fixing" ([0019](docs/adr/0019-intl-is-optional.md)). - Numbers shown to users go through `%F`, never `%f`: `sprintf()`'s `%f` follows `LC_NUMERIC` and prints a decimal comma on a French server. +- `TranslationFileScanner` keeps each parsed file in the pool + `cyllene_digital_ai_translation.translation_files_cache` (adapter `cache.app`, 7 days), keyed + by its path, date of modification, size and `CACHE_FORMAT`: a change to what `parseFile()` + returns bumps `CACHE_FORMAT`, or hosts are served the old shape until their entries expire. + The walk of the directories is never cached, and the scanner is reset between two Messenger + messages ([0027](docs/adr/0027-parsed-files-cached-by-fingerprint.md)). The test kernels + share that pool on the filesystem: a test rewriting a fixture in place within the same second + and at the same size reads the previous entry. - `TranslationSuggestionRepository::transactional()` is hand-made on purpose: `EntityManager::wrapInTransaction()` closes the host's entity manager on any exception, a refused approval included ([0015](docs/adr/0015-hand-made-transactions.md)). diff --git a/config/definition.php b/config/definition.php index 838ca3a..0a33441 100644 --- a/config/definition.php +++ b/config/definition.php @@ -69,6 +69,10 @@ ->min(0) ->defaultNull() ->end() + ->booleanNode('translation_files_cache') + ->info('Keeps each parsed translation file in the "cyllene_digital_ai_translation.translation_files_cache" pool (adapter cache.app, 7 days, both overridable under framework.cache.pools), keyed by its path, date of modification and size: a request stats the files instead of parsing them. false = parse on every new process.') + ->defaultTrue() + ->end() ->scalarNode('default_locale') ->info('The project\'s default locale: the coverage reference and the default generation source. Falls back to the framework default_locale matched against the available locales, then to the first available locale.') ->defaultNull() diff --git a/docs/adr/0027-parsed-files-cached-by-fingerprint.md b/docs/adr/0027-parsed-files-cached-by-fingerprint.md new file mode 100644 index 0000000..f64e10c --- /dev/null +++ b/docs/adr/0027-parsed-files-cached-by-fingerprint.md @@ -0,0 +1,42 @@ +# 0027. Parsed translation files are cached by their fingerprint, never the walk of the directories + +- Status: accepted +- Date: 2026-10-15 + +## Context + +The bundle parses the host's translation files itself, since the translator's compiled +catalogues mix in messages it does not expose. A view over every catalogue parses every file, +in every new process: under PHP-FPM, on every request. Files change without any event the +bundle could listen to: a deployment, an edit in development, a theme package updated. + +## Decision + +Each parsed file is kept in a cache pool, under a key made of its path, its date of +modification, its size and a format version. Nothing invalidates an entry: a file that changes +gets a new key, and the pool's lifetime removes what is left behind. The walk of the +directories is not cached, so a file added or removed is seen at once. The cache is on by +default, the pool is the bundle's own, built on `cache.app`, and the scanner's in-memory memo +is reset between two Messenger messages. + +## Alternatives rejected + +- **A cache emptied at deployment**: no file access at all, but a file edited in place is never + seen, which is the usual case in development. +- **Symfony's `ConfigCache`**, as the translator does: resources are only checked in debug, so + production has the same blind spot, for more code written into the cache directory. +- **The date of modification alone**: two writes within the same second would keep the first + one; the size tells most of them apart. + +## Consequences + +- A request stats every file of the catalogues it reads instead of parsing them. +- A file rewritten within the same second with the same size keeps the previous entry, which + the documentation says, with the command that empties the pool. +- A shared pool needs nothing more on several servers: an entry only matches the file it was + written for. +- What the scanner stores changes only with a new format version in the key. + +## Related + +- [0001. Overrides are a database delta, the translation files are never written](0001-overrides-are-a-database-delta.md) diff --git a/docs/adr/README.md b/docs/adr/README.md index dce22c1..132d581 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -46,3 +46,4 @@ Alternatives rejected when there were some, Consequences, Related) and a line in | 0024 | [The keys of a batch are reserved before the provider is called](0024-keys-reserved-before-the-provider-is-called.md) | | 0025 | [A deleted scope is purged in a transaction of its own, its journal kept](0025-a-deleted-scope-is-purged-journal-kept.md) | | 0026 | [A new capability asked of the host is a separate optional interface](0026-new-host-capabilities-are-optional-interfaces.md) | +| 0027 | [Parsed translation files are cached by their fingerprint, never the walk of the directories](0027-parsed-files-cached-by-fingerprint.md) | diff --git a/docs/configuration-reference.md b/docs/configuration-reference.md index 98e10e2..6f505cc 100644 --- a/docs/configuration-reference.md +++ b/docs/configuration-reference.md @@ -32,6 +32,13 @@ cyllene_digital_ai_translation: # coverage command always recounts from scratch, whatever this setting. coverage_cache_ttl: null + # Keeps each parsed translation file in the + # "cyllene_digital_ai_translation.translation_files_cache" pool, keyed by its + # path, date of modification and size: a new process stats the files instead + # of parsing them again. false = parse them in every new process. See + # operations/cache.md. + translation_files_cache: true + # Additional translation roots (label => directory), browsed as "@label/…" # catalogues, e.g. the translations/ of theme packages. Within a domain, a # "@label" override takes precedence over the main root's (as theme files do @@ -138,6 +145,22 @@ cyllene_digital_ai_translation: the unknown name then fails on the first generation, with the list of available providers. Left unset, it falls back to the first entry of `providers`, or to the custom provider when it is the only one. +- **`translation_files_cache`** stores into the pool + `cyllene_digital_ai_translation.translation_files_cache`, which the bundle declares with + `adapter: cache.app` and `default_lifetime: 604800` (7 days). A host entry for the same pool + under `framework.cache.pools` is merged over it, so either can be changed there: + + ```yaml + # config/packages/framework.yaml + framework: + cache: + pools: + cyllene_digital_ai_translation.translation_files_cache: + adapter: cache.adapter.php_files # read back through opcache + ``` + + `bin/console cache:pool:clear cyllene_digital_ai_translation.translation_files_cache` + empties it alone. - **`providers` may stay empty**: the override workflow works without AI; the interactive console's generation entry explains that no provider is configured, and a host surface listing suggestions can show its own warning. diff --git a/docs/operations/cache.md b/docs/operations/cache.md index 57c81eb..3f4d0a0 100644 --- a/docs/operations/cache.md +++ b/docs/operations/cache.md @@ -2,7 +2,8 @@ Two layers sit between a stored override and the text on screen. Both are invalidated **automatically** on every save/remove; this page is mostly here to explain what happens and -to know when to step in manually. +to know when to step in manually. A third one, under the bundle's own screens and commands, +keeps the parsed translation files: see [The parsed translation files](#the-parsed-translation-files). ## The layers @@ -37,6 +38,29 @@ query per (locale, scope) per hour). - **Direct writes bypassing `OverrideWriter`** (raw SQL, a fixture): no event is dispatched, so the in-memory map of an already-running process is not dropped either (see the entry above). +## The parsed translation files + +The bundle reads the host's translation files itself, to list the catalogues, compare the +locales and find the missing keys: the translator's compiled catalogues mix in the framework's +and the other bundles' messages. Parsing them is most of the time of a view over every +catalogue, so each parsed file is kept in the pool +`cyllene_digital_ai_translation.translation_files_cache` (`translation_files_cache: true`, the +default). + +| What | How | +| --- | --- | +| Key | The file's path, its date of modification and its size, and a format version the bundle bumps when it changes what it stores | +| Invalidation | None needed: a file that changes gets a new key, the old entry is orphaned | +| Lifetime | 7 days (`default_lifetime` of the pool), which bounds the orphaned entries | +| Not cached | The walk of the directories: a file added or removed is seen on the next request | +| Long-running processes | The scanner is reset between two Messenger messages (`kernel.reset`): a worker sees an edited file without a restart | + +A file rewritten within the same second **with the same size** keeps the previous entry: touch +it, or empty the pool with +`bin/console cache:pool:clear cyllene_digital_ai_translation.translation_files_cache`. Changing +the adapter or the lifetime: see the [configuration reference](../configuration-reference.md#notes). +To parse the files in every new process instead, set `translation_files_cache: false`. + ## What the bundle does not touch The HTTP cache, Varnish, or any application-side Twig fragment cache: an overridden label on an diff --git a/docs/operations/multi-server.md b/docs/operations/multi-server.md index bf1800d..ddff52c 100644 --- a/docs/operations/multi-server.md +++ b/docs/operations/multi-server.md @@ -34,6 +34,11 @@ workers only see the change once their `cache.app` entry is gone. The same applies to the coverage report cache (`coverage_cache_ttl`), which lives in the same pool. +The parsed translation files (`translation_files_cache`) are kept in a pool of their own, +built on `cache.app` by default. Shared or not, nothing goes wrong: the key holds each +file's path, date of modification and size, so a node only reads entries written for the file +it has, and releases deployed under different paths simply miss each other's entries. + Symfony's compiled catalogues (`kernel.cache_dir/translations`) are node-local too, but the bundle never touches them: they hold the file translations, never the overrides, which are applied on top at runtime. diff --git a/docs/public-contract.md b/docs/public-contract.md index 06a1cad..86b3b10 100644 --- a/docs/public-contract.md +++ b/docs/public-contract.md @@ -26,6 +26,8 @@ page name against the code: a rename goes red until the page, and `UPGRADE.md`, - **The configuration tree** under `cyllene_digital_ai_translation`: every key of the [configuration reference](configuration-reference.md), its type and its default. +- **The cache pool `cyllene_digital_ai_translation.translation_files_cache`**: the name a + host's `framework.cache.pools` entry uses to change its adapter or lifetime. - **The Messenger transport `cyllene_ai_translation`**, and the routing of `GenerateSuggestionsMessage` to it: what a host's `messenger:consume` and an integration's worker configuration name. diff --git a/src/Catalogue/TranslationFileScanner.php b/src/Catalogue/TranslationFileScanner.php index 70ee057..de9ecf1 100644 --- a/src/Catalogue/TranslationFileScanner.php +++ b/src/Catalogue/TranslationFileScanner.php @@ -10,6 +10,8 @@ use Symfony\Component\Translation\Loader\LoaderInterface; use Symfony\Component\Translation\Loader\XliffFileLoader; use Symfony\Component\Translation\Loader\YamlFileLoader; +use Symfony\Contracts\Cache\CacheInterface; +use Symfony\Contracts\Service\ResetInterface; /** * Reads the host project's translations/ directory — the single source of the @@ -23,14 +25,21 @@ * getIgnoredFiles(). * * Everything is memoized for the lifetime of the service (one request / one CLI run): - * browsing triggers several reads of the same catalogue in a row. + * browsing triggers several reads of the same catalogue in a row. Under that memo, each + * parsed file is kept in a persistent pool (translation_files_cache) keyed by its path, date of modification and + * size, so a new process stats the files instead of parsing them again. The walk of the + * directories is never cached: a file added or removed is seen at once. The memo is + * dropped between two Messenger messages (kernel.reset), so a worker sees an edited file. */ -final class TranslationFileScanner +final class TranslationFileScanner implements ResetInterface { private const array SUPPORTED_EXTENSIONS = ['yaml', 'yml', 'xlf', 'xliff', 'json']; private const string INTL_ICU_SUFFIX = '+intl-icu'; + /** Part of every persistent cache key: bump it when what loadFile() returns changes. */ + private const int CACHE_FORMAT = 1; + /** @var array|null identifier => catalogue, sorted */ private ?array $catalogues = null; @@ -57,15 +66,26 @@ final class TranslationFileScanner /** * @param array $additionalPaths label => directory, browsed as "@label/…" + * @param CacheInterface|null $filesCache null when translation_files_cache is off */ public function __construct( #[Autowire(param: 'cyllene_digital_ai_translation.translations_path')] private readonly string $translationsPath, #[Autowire(param: 'cyllene_digital_ai_translation.additional_paths')] private readonly array $additionalPaths = [], + private readonly ?CacheInterface $filesCache = null, ) { } + public function reset(): void + { + $this->catalogues = null; + $this->files = []; + $this->ignoredFiles = []; + $this->messagesCache = []; + $this->fileCache = []; + } + /** * @return list */ @@ -331,13 +351,51 @@ private function loadFile(array $file): array return $this->fileCache[$file['path']]; } + $parsed = null; + if (null !== $this->filesCache && null !== $key = $this->persistentKey($file['path'])) { + $parsed = $this->filesCache->get($key, fn (): array => $this->parseFile($file)); + } + $parsed ??= $this->parseFile($file); + + // Recorded on every load, the cached ones included: an unparseable file stays + // reported for as long as it is unparseable, not only by the process that parsed it. + if (null !== $parsed['error']) { + $this->ignoredFiles[$file['path']] = $parsed['error']; + } + + return $this->fileCache[$file['path']] = $parsed['messages']; + } + + /** + * Two writes within the same second rarely leave the same size, and paths differing + * between releases only miss the cache. Null when the file cannot be stat'ed: it is then + * parsed, and fails, as if there were no cache. + */ + private function persistentKey(string $path): ?string + { + // A long-running process would otherwise be answered from PHP's stat cache. + clearstatcache(true, $path); + $stat = @stat($path); + + if (false === $stat) { + return null; + } + + return 'file_'.hash('xxh128', self::CACHE_FORMAT."\0".$path."\0".$stat['mtime']."\0".$stat['size']); + } + + /** + * @param array{path: string, locale: string, intlIcu: bool, extension: string} $file + * + * @return array{messages: array, error: ?string} + */ + private function parseFile(array $file): array + { try { $catalogue = $this->getLoader($file['extension'])->load($file['path'], $file['locale'], 'messages'); } catch (\Throwable $e) { - $this->ignoredFiles[$file['path']] = \sprintf('The file could not be parsed: %s', $e->getMessage()); - - // Memoised too: an unparseable file must not be retried on every lookup. - return $this->fileCache[$file['path']] = []; + // Kept like a parsed file: an unparseable file must not be retried on every lookup. + return ['messages' => [], 'error' => \sprintf('The file could not be parsed: %s', $e->getMessage())]; } $messages = []; @@ -347,7 +405,7 @@ private function loadFile(array $file): array } } - return $this->fileCache[$file['path']] = $messages; + return ['messages' => $messages, 'error' => null]; } private function getLoader(string $extension): LoaderInterface diff --git a/src/CylleneDigitalAiTranslationBundle.php b/src/CylleneDigitalAiTranslationBundle.php index 55be7ed..8bfe4ba 100644 --- a/src/CylleneDigitalAiTranslationBundle.php +++ b/src/CylleneDigitalAiTranslationBundle.php @@ -9,6 +9,7 @@ use CylleneDigital\AiTranslationBundle\Bridge\DeeplProvider; use CylleneDigital\AiTranslationBundle\Bridge\DefaultProvider; use CylleneDigital\AiTranslationBundle\Bridge\OpenAiProvider; +use CylleneDigital\AiTranslationBundle\Catalogue\TranslationFileScanner; use CylleneDigital\AiTranslationBundle\DependencyInjection\DefaultProviderPass; use CylleneDigital\AiTranslationBundle\Estimation\ModelPriceProvider; use CylleneDigital\AiTranslationBundle\Message\GenerateSuggestionsMessage; @@ -41,6 +42,11 @@ final class CylleneDigitalAiTranslationBundle extends AbstractBundle */ private const string DOCTRINE_TRANSPORT_FACTORY = 'Symfony\\Component\\Messenger\\Bridge\\Doctrine\\Transport\\DoctrineTransportFactory'; + private const string FILES_CACHE_POOL = 'cyllene_digital_ai_translation.translation_files_cache'; + + /** An entry is orphaned, never deleted, when its file changes: the lifetime bounds them. */ + private const int FILES_CACHE_LIFETIME = 604800; + protected string $extensionAlias = 'cyllene_digital_ai_translation'; public function getPath(): string @@ -59,7 +65,7 @@ public function configure(DefinitionConfigurator $definition): void } /** - * @param array{default_provider: ?string, translations_path: string, additional_paths: array, default_locale: ?string, coverage_cache_ttl: ?int, llm_context: ?string, model_prices_url: string|false, generation_lock_ttl: int, messenger_transport_dsn: string, providers: array} $config + * @param array{default_provider: ?string, translations_path: string, additional_paths: array, default_locale: ?string, coverage_cache_ttl: ?int, translation_files_cache: bool, llm_context: ?string, model_prices_url: string|false, generation_lock_ttl: int, messenger_transport_dsn: string, providers: array} $config */ public function loadExtension(array $config, ContainerConfigurator $container, ContainerBuilder $builder): void { @@ -74,6 +80,9 @@ public function loadExtension(array $config, ContainerConfigurator $container, C ->set('cyllene_digital_ai_translation.model_prices_url', $config['model_prices_url']) ->set('cyllene_digital_ai_translation.default_provider', $config['default_provider'] ?? array_key_first($config['providers'])); + $builder->getDefinition(TranslationFileScanner::class) + ->setArgument('$filesCache', $config['translation_files_cache'] ? new Reference(self::FILES_CACHE_POOL) : null); + // A retrying client shared by all bridges: rate limits (429), an unavailable // provider (503) and Anthropic's "overloaded" (529) are retried twice with exponential backoff before failing the // catalogue being translated. Only codes saying the request was NOT processed: @@ -137,6 +146,22 @@ public function prependExtension(ContainerConfigurator $container, ContainerBuil ]); } + // Declared whether translation_files_cache is on or not (prepend runs before the + // config is processed): an unused pool costs nothing. A host's own entry for the + // same pool is merged over this one, so it can change the adapter or the lifetime. + if ($builder->hasExtension('framework')) { + $builder->prependExtensionConfig('framework', [ + 'cache' => [ + 'pools' => [ + self::FILES_CACHE_POOL => [ + 'adapter' => 'cache.app', + 'default_lifetime' => self::FILES_CACHE_LIFETIME, + ], + ], + ], + ]); + } + if ($builder->hasExtension('doctrine')) { $builder->prependExtensionConfig('doctrine', [ 'orm' => [ diff --git a/tests/App/FullConfigTestKernel.php b/tests/App/FullConfigTestKernel.php index b37ebf3..3f1774a 100644 --- a/tests/App/FullConfigTestKernel.php +++ b/tests/App/FullConfigTestKernel.php @@ -35,6 +35,7 @@ protected function configureContainer(ContainerConfigurator $container): void ], 'default_locale' => 'en', 'coverage_cache_ttl' => 60, + 'translation_files_cache' => false, 'llm_context' => 'An online shop.', 'model_prices_url' => false, 'generation_lock_ttl' => 600, diff --git a/tests/Functional/TranslationFilesCacheWiringTest.php b/tests/Functional/TranslationFilesCacheWiringTest.php new file mode 100644 index 0000000..1c037fa --- /dev/null +++ b/tests/Functional/TranslationFilesCacheWiringTest.php @@ -0,0 +1,66 @@ +get('cyllene_digital_ai_translation.translation_files_cache'), + $this->filesCache($this->scanner()), + ); + } + + public function testTheScannerHasNoPoolWhenTheCacheIsOff(): void + { + static::$class = FullConfigTestKernel::class; + self::bootKernel(); + + self::assertNull($this->filesCache($this->scanner())); + } + + public function testTheScannerIsResetWithTheOtherServices(): void + { + static::$class = TestKernel::class; + self::bootKernel(); + + $scanner = $this->scanner(); + $scanner->getCatalogues(); + + $resetter = self::getContainer()->get('services_resetter'); + self::assertInstanceOf(ResetInterface::class, $resetter); + $resetter->reset(); + + self::assertNull((new \ReflectionProperty(TranslationFileScanner::class, 'catalogues'))->getValue($scanner), 'The next call walks the directories again.'); + } + + private function scanner(): TranslationFileScanner + { + $scanner = self::getContainer()->get(TranslationFileScanner::class); + self::assertInstanceOf(TranslationFileScanner::class, $scanner); + + return $scanner; + } + + private function filesCache(TranslationFileScanner $scanner): mixed + { + return (new \ReflectionProperty(TranslationFileScanner::class, 'filesCache'))->getValue($scanner); + } +} diff --git a/tests/Unit/Catalogue/TranslationFileScannerCacheTest.php b/tests/Unit/Catalogue/TranslationFileScannerCacheTest.php new file mode 100644 index 0000000..865cc00 --- /dev/null +++ b/tests/Unit/Catalogue/TranslationFileScannerCacheTest.php @@ -0,0 +1,108 @@ +dir = sys_get_temp_dir().'/cyllene_scanner_cache_'.bin2hex(random_bytes(4)); + mkdir($this->dir, 0o777, true); + $this->pool = new ArrayAdapter(); + } + + protected function tearDown(): void + { + (new Filesystem())->remove($this->dir); + } + + public function testANewProcessIsServedTheFileWithoutParsingItAgain(): void + { + $this->write('messages.fr.json', '{"app.title":"one"}'); + self::assertSame(['app.title' => 'one'], $this->scanner()->getMessages('messages', 'fr')); + + // Same date, same size, other content: only a cached read still says "one". + $this->write('messages.fr.json', '{"app.title":"two"}'); + + self::assertSame(['app.title' => 'one'], $this->scanner()->getMessages('messages', 'fr')); + self::assertSame(['app.title' => 'two'], (new TranslationFileScanner($this->dir))->getMessages('messages', 'fr')); + } + + public function testAFileWhoseDateChangedIsParsedAgain(): void + { + $this->write('messages.fr.json', '{"app.title":"one"}'); + $this->scanner()->getMessages('messages', 'fr'); + + $this->write('messages.fr.json', '{"app.title":"two"}', self::MTIME + 1); + + self::assertSame(['app.title' => 'two'], $this->scanner()->getMessages('messages', 'fr')); + } + + /** Two writes within the same second: the size tells them apart. */ + public function testAFileWhoseSizeChangedIsParsedAgain(): void + { + $this->write('messages.fr.json', '{"app.title":"one"}'); + $this->scanner()->getMessages('messages', 'fr'); + + $this->write('messages.fr.json', '{"app.title":"three"}'); + + self::assertSame(['app.title' => 'three'], $this->scanner()->getMessages('messages', 'fr')); + } + + public function testAnUnparseableFileIsStillReportedWhenServedFromTheCache(): void + { + $this->write('messages.fr.json', '{"app.title":'); + $this->scanner()->getMessages('messages', 'fr'); + + $scanner = $this->scanner(); + + self::assertSame([], $scanner->getMessages('messages', 'fr')); + self::assertStringStartsWith('The file could not be parsed', $scanner->getIgnoredFiles()[$this->dir.'/messages.fr.json'] ?? ''); + } + + /** What a Messenger worker gets between two messages (kernel.reset). */ + public function testAResetScannerSeesAnEditedFileAndANewOne(): void + { + $this->write('messages.fr.json', '{"app.title":"one"}'); + $scanner = $this->scanner(); + $scanner->getMessages('messages', 'fr'); + + $this->write('messages.fr.json', '{"app.title":"two"}', self::MTIME + 1); + $this->write('messages.de.json', '{"app.title":"eins"}'); + self::assertSame(['app.title' => 'one'], $scanner->getMessages('messages', 'fr'), 'Memoized until the reset.'); + + $scanner->reset(); + + self::assertSame(['app.title' => 'two'], $scanner->getMessages('messages', 'fr')); + self::assertSame(['de', 'fr'], $scanner->getLocales()); + } + + private function scanner(): TranslationFileScanner + { + return new TranslationFileScanner($this->dir, filesCache: $this->pool); + } + + private function write(string $name, string $content, int $mtime = self::MTIME): void + { + file_put_contents($this->dir.'/'.$name, $content); + touch($this->dir.'/'.$name, $mtime); + } +} From 65d3012e6a023bd0889446b81209bef7f36c011b Mon Sep 17 00:00:00 2001 From: Julien Dehais Date: Thu, 15 Oct 2026 08:20:18 +0200 Subject: [PATCH 2/5] docs: rewrap the scanner docblock and reword the persistentKey() one --- src/Catalogue/TranslationFileScanner.php | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/src/Catalogue/TranslationFileScanner.php b/src/Catalogue/TranslationFileScanner.php index de9ecf1..729d0d9 100644 --- a/src/Catalogue/TranslationFileScanner.php +++ b/src/Catalogue/TranslationFileScanner.php @@ -26,10 +26,11 @@ * * Everything is memoized for the lifetime of the service (one request / one CLI run): * browsing triggers several reads of the same catalogue in a row. Under that memo, each - * parsed file is kept in a persistent pool (translation_files_cache) keyed by its path, date of modification and - * size, so a new process stats the files instead of parsing them again. The walk of the - * directories is never cached: a file added or removed is seen at once. The memo is - * dropped between two Messenger messages (kernel.reset), so a worker sees an edited file. + * parsed file is kept in a persistent pool (translation_files_cache), keyed by its path, + * date of modification and size, so a new process stats the files instead of parsing them + * again. The walk of the directories is never cached: a file added or removed is seen at + * once. The memo is dropped between two Messenger messages (kernel.reset), so a worker + * sees an edited file. */ final class TranslationFileScanner implements ResetInterface { @@ -368,8 +369,8 @@ private function loadFile(array $file): array /** * Two writes within the same second rarely leave the same size, and paths differing - * between releases only miss the cache. Null when the file cannot be stat'ed: it is then - * parsed, and fails, as if there were no cache. + * between releases only miss the cache. Null when the file's metadata cannot be read: it + * is then parsed, and fails, as if there were no cache. */ private function persistentKey(string $path): ?string { From 729e5505fae9011adbf037222883a110330938ca Mon Sep 17 00:00:00 2001 From: Julien Dehais Date: Thu, 15 Oct 2026 09:08:56 +0200 Subject: [PATCH 3/5] docs: say that cache:clear does not empty the files pool when APP_SHARE_DIR is set --- docs/operations/cache.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/operations/cache.md b/docs/operations/cache.md index 3f4d0a0..2cf4270 100644 --- a/docs/operations/cache.md +++ b/docs/operations/cache.md @@ -57,9 +57,11 @@ default). A file rewritten within the same second **with the same size** keeps the previous entry: touch it, or empty the pool with -`bin/console cache:pool:clear cyllene_digital_ai_translation.translation_files_cache`. Changing -the adapter or the lifetime: see the [configuration reference](../configuration-reference.md#notes). -To parse the files in every new process instead, set `translation_files_cache: false`. +`bin/console cache:pool:clear cyllene_digital_ai_translation.translation_files_cache`. +`cache:clear` is not enough when `APP_SHARE_DIR` is set, as the Symfony 7.4 and 8 skeletons do +(`var/share`): the filesystem `cache.app`, and this pool with it, then lives in that directory, +which `cache:clear` leaves alone. Changing the adapter or the lifetime: see the +[configuration reference](../configuration-reference.md#notes). To parse the files in every new process instead, set `translation_files_cache: false`. ## What the bundle does not touch From b691563afb4736c452c8318bde99a5b48f957a10 Mon Sep 17 00:00:00 2001 From: Julien Dehais Date: Thu, 15 Oct 2026 14:38:14 +0200 Subject: [PATCH 4/5] feat: key the parsed translation files by their content, and never cache an unreadable one The date of modification and the size missed a file rewritten within the same second at the same size, and a release deployed under the same path, with the dates a build froze, on a pool shared between releases. Reading and hashing a file costs a fraction of reading its cache entry. A file that cannot be read is now reported by the scanner itself (the JSON loader read it as empty) and never cached: a chmod changes nothing a key could see. --- CLAUDE.md | 7 ++- config/definition.php | 2 +- ...0027-parsed-files-cached-by-fingerprint.md | 23 +++++---- docs/concepts/catalogues.md | 4 +- docs/configuration-reference.md | 4 +- docs/operations/cache.md | 9 ++-- docs/operations/multi-server.md | 4 +- src/Catalogue/TranslationFileScanner.php | 28 +++++++---- .../TranslationFileScannerCacheTest.php | 50 ++++++++++++------- 9 files changed, 74 insertions(+), 57 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 30de534..9ce780e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -157,12 +157,11 @@ facade - is in the records 0016, 0017, 0022 and 0023: read them before "fixing" and prints a decimal comma on a French server. - `TranslationFileScanner` keeps each parsed file in the pool `cyllene_digital_ai_translation.translation_files_cache` (adapter `cache.app`, 7 days), keyed - by its path, date of modification, size and `CACHE_FORMAT`: a change to what `parseFile()` + by its path, a hash of its content and `CACHE_FORMAT`: a change to what `parseFile()` returns bumps `CACHE_FORMAT`, or hosts are served the old shape until their entries expire. The walk of the directories is never cached, and the scanner is reset between two Messenger - messages ([0027](docs/adr/0027-parsed-files-cached-by-fingerprint.md)). The test kernels - share that pool on the filesystem: a test rewriting a fixture in place within the same second - and at the same size reads the previous entry. + messages. A file that cannot be read is reported, never cached + ([0027](docs/adr/0027-parsed-files-cached-by-fingerprint.md)). - `TranslationSuggestionRepository::transactional()` is hand-made on purpose: `EntityManager::wrapInTransaction()` closes the host's entity manager on any exception, a refused approval included ([0015](docs/adr/0015-hand-made-transactions.md)). diff --git a/config/definition.php b/config/definition.php index 0a33441..ba14992 100644 --- a/config/definition.php +++ b/config/definition.php @@ -70,7 +70,7 @@ ->defaultNull() ->end() ->booleanNode('translation_files_cache') - ->info('Keeps each parsed translation file in the "cyllene_digital_ai_translation.translation_files_cache" pool (adapter cache.app, 7 days, both overridable under framework.cache.pools), keyed by its path, date of modification and size: a request stats the files instead of parsing them. false = parse on every new process.') + ->info('Keeps each parsed translation file in the "cyllene_digital_ai_translation.translation_files_cache" pool (adapter cache.app, 7 days, both overridable under framework.cache.pools), keyed by its path and a hash of its content: a request reads and hashes the files instead of parsing them. false = parse on every new process.') ->defaultTrue() ->end() ->scalarNode('default_locale') diff --git a/docs/adr/0027-parsed-files-cached-by-fingerprint.md b/docs/adr/0027-parsed-files-cached-by-fingerprint.md index f64e10c..8bb5e61 100644 --- a/docs/adr/0027-parsed-files-cached-by-fingerprint.md +++ b/docs/adr/0027-parsed-files-cached-by-fingerprint.md @@ -12,10 +12,10 @@ bundle could listen to: a deployment, an edit in development, a theme package up ## Decision -Each parsed file is kept in a cache pool, under a key made of its path, its date of -modification, its size and a format version. Nothing invalidates an entry: a file that changes -gets a new key, and the pool's lifetime removes what is left behind. The walk of the -directories is not cached, so a file added or removed is seen at once. The cache is on by +Each parsed file is kept in a cache pool, under a key made of its path, a hash of its content +and a format version. Nothing invalidates an entry: a file that changes gets a new key, and the pool's lifetime removes what is left behind. The walk of the +directories is not cached, so a file added or removed is seen at once. A file that cannot be +read is reported without being cached: making it readable changes nothing in its key. The cache is on by default, the pool is the bundle's own, built on `cache.app`, and the scanner's in-memory memo is reset between two Messenger messages. @@ -25,16 +25,17 @@ is reset between two Messenger messages. seen, which is the usual case in development. - **Symfony's `ConfigCache`**, as the translator does: resources are only checked in debug, so production has the same blind spot, for more code written into the cache directory. -- **The date of modification alone**: two writes within the same second would keep the first - one; the size tells most of them apart. +- **The date of modification and the size**: no read of the file, but a file rewritten within + the same second at the same size keeps the previous entry, and so does a release deployed + under the same path, with the dates a build froze, on a pool shared between releases. + Reading and hashing a file costs a fraction of reading its cache entry, and far less than + parsing it. ## Consequences -- A request stats every file of the catalogues it reads instead of parsing them. -- A file rewritten within the same second with the same size keeps the previous entry, which - the documentation says, with the command that empties the pool. -- A shared pool needs nothing more on several servers: an entry only matches the file it was - written for. +- A request reads and hashes every file of the catalogues it reads instead of parsing them. +- A shared pool needs nothing more, on several servers or between releases: an entry only + matches the content it was parsed from. - What the scanner stores changes only with a new format version in the key. ## Related diff --git a/docs/concepts/catalogues.md b/docs/concepts/catalogues.md index f5828ed..ed947f2 100644 --- a/docs/concepts/catalogues.md +++ b/docs/concepts/catalogues.md @@ -30,8 +30,8 @@ the ICU variant taking precedence (as in the translator). A locale written with translation tools name their files (`messages.pt-BR.yaml`), is read with `_`: it is the `pt_BR` locale, the spelling Symfony and the stored overrides use. -**Nothing is silently ignored**: a file whose name or content cannot be interpreted is listed -with its reason (`getIgnoredFiles()`), for the caller to surface as a warning. +**Nothing is silently ignored**: a file that cannot be read, or whose name or content cannot be +interpreted, is listed with its reason (`getIgnoredFiles()`), for the caller to surface as a warning. The scanned directory is configured via `translations_path` (default: `%kernel.project_dir%/translations`). `additional_paths` adds more roots (a theme, a diff --git a/docs/configuration-reference.md b/docs/configuration-reference.md index 6f505cc..3166b2c 100644 --- a/docs/configuration-reference.md +++ b/docs/configuration-reference.md @@ -34,8 +34,8 @@ cyllene_digital_ai_translation: # Keeps each parsed translation file in the # "cyllene_digital_ai_translation.translation_files_cache" pool, keyed by its - # path, date of modification and size: a new process stats the files instead - # of parsing them again. false = parse them in every new process. See + # path and a hash of its content: a new process reads and hashes the files + # instead of parsing them again. false = parse them in every new process. See # operations/cache.md. translation_files_cache: true diff --git a/docs/operations/cache.md b/docs/operations/cache.md index 2cf4270..7f05e51 100644 --- a/docs/operations/cache.md +++ b/docs/operations/cache.md @@ -49,14 +49,13 @@ default). | What | How | | --- | --- | -| Key | The file's path, its date of modification and its size, and a format version the bundle bumps when it changes what it stores | -| Invalidation | None needed: a file that changes gets a new key, the old entry is orphaned | +| Key | The file's path, a hash of its content, and a format version the bundle bumps when it changes what it stores | +| Invalidation | None needed: a file whose content changes gets a new key, the old entry is orphaned | | Lifetime | 7 days (`default_lifetime` of the pool), which bounds the orphaned entries | -| Not cached | The walk of the directories: a file added or removed is seen on the next request | +| Not cached | The walk of the directories: a file added or removed is seen on the next request. A file that cannot be read: once readable again, it is read on the next request | | Long-running processes | The scanner is reset between two Messenger messages (`kernel.reset`): a worker sees an edited file without a restart | -A file rewritten within the same second **with the same size** keeps the previous entry: touch -it, or empty the pool with +To empty the pool: `bin/console cache:pool:clear cyllene_digital_ai_translation.translation_files_cache`. `cache:clear` is not enough when `APP_SHARE_DIR` is set, as the Symfony 7.4 and 8 skeletons do (`var/share`): the filesystem `cache.app`, and this pool with it, then lives in that directory, diff --git a/docs/operations/multi-server.md b/docs/operations/multi-server.md index ddff52c..28cd7ef 100644 --- a/docs/operations/multi-server.md +++ b/docs/operations/multi-server.md @@ -36,8 +36,8 @@ pool. The parsed translation files (`translation_files_cache`) are kept in a pool of their own, built on `cache.app` by default. Shared or not, nothing goes wrong: the key holds each -file's path, date of modification and size, so a node only reads entries written for the file -it has, and releases deployed under different paths simply miss each other's entries. +file's path and a hash of its content, so a node only reads entries parsed from the content it +has, between releases too, whatever the dates of modification a build leaves. Symfony's compiled catalogues (`kernel.cache_dir/translations`) are node-local too, but the bundle never touches them: they hold the file translations, never the overrides, which are diff --git a/src/Catalogue/TranslationFileScanner.php b/src/Catalogue/TranslationFileScanner.php index 729d0d9..b4196ef 100644 --- a/src/Catalogue/TranslationFileScanner.php +++ b/src/Catalogue/TranslationFileScanner.php @@ -26,9 +26,9 @@ * * Everything is memoized for the lifetime of the service (one request / one CLI run): * browsing triggers several reads of the same catalogue in a row. Under that memo, each - * parsed file is kept in a persistent pool (translation_files_cache), keyed by its path, - * date of modification and size, so a new process stats the files instead of parsing them - * again. The walk of the directories is never cached: a file added or removed is seen at + * parsed file is kept in a persistent pool (translation_files_cache), keyed by its path + * and a hash of its content, so a new process reads and hashes the files instead of parsing + * them again. The walk of the directories is never cached: a file added or removed is seen at * once. The memo is dropped between two Messenger messages (kernel.reset), so a worker * sees an edited file. */ @@ -352,6 +352,14 @@ private function loadFile(array $file): array return $this->fileCache[$file['path']]; } + // Checked here, not left to the loaders: the JSON one reads an unreadable file as empty. + // Never cached: a chmod or a chown changes nothing a cache key could see. + if (!is_readable($file['path'])) { + $this->ignoredFiles[$file['path']] = 'The file cannot be read: check its permissions.'; + + return $this->fileCache[$file['path']] = []; + } + $parsed = null; if (null !== $this->filesCache && null !== $key = $this->persistentKey($file['path'])) { $parsed = $this->filesCache->get($key, fn (): array => $this->parseFile($file)); @@ -368,21 +376,19 @@ private function loadFile(array $file): array } /** - * Two writes within the same second rarely leave the same size, and paths differing - * between releases only miss the cache. Null when the file's metadata cannot be read: it - * is then parsed, and fails, as if there were no cache. + * The content, not the date of modification: a build may freeze the dates, and a pool + * shared between releases deployed under the same path would serve the previous content. + * Null when the file cannot be read: it is then parsed as if there were no cache. */ private function persistentKey(string $path): ?string { - // A long-running process would otherwise be answered from PHP's stat cache. - clearstatcache(true, $path); - $stat = @stat($path); + $content = @file_get_contents($path); - if (false === $stat) { + if (false === $content) { return null; } - return 'file_'.hash('xxh128', self::CACHE_FORMAT."\0".$path."\0".$stat['mtime']."\0".$stat['size']); + return 'file_'.hash('xxh128', self::CACHE_FORMAT."\0".$path."\0".$content); } /** diff --git a/tests/Unit/Catalogue/TranslationFileScannerCacheTest.php b/tests/Unit/Catalogue/TranslationFileScannerCacheTest.php index 865cc00..1289a23 100644 --- a/tests/Unit/Catalogue/TranslationFileScannerCacheTest.php +++ b/tests/Unit/Catalogue/TranslationFileScannerCacheTest.php @@ -11,13 +11,11 @@ /** * The parsed files outlive the process (translation_files_cache): a new scanner on the same - * pool reads a file again only when its date of modification or its size changed. Each - * "new process" below is a new scanner instance on a shared pool. + * pool reads a file again only when its content changed. Each "new process" below is a new + * scanner instance on a shared pool. */ final class TranslationFileScannerCacheTest extends TestCase { - private const int MTIME = 1_760_000_000; - private string $dir; private ArrayAdapter $pool; @@ -37,34 +35,49 @@ protected function tearDown(): void public function testANewProcessIsServedTheFileWithoutParsingItAgain(): void { $this->write('messages.fr.json', '{"app.title":"one"}'); - self::assertSame(['app.title' => 'one'], $this->scanner()->getMessages('messages', 'fr')); + $this->scanner()->getMessages('messages', 'fr'); - // Same date, same size, other content: only a cached read still says "one". - $this->write('messages.fr.json', '{"app.title":"two"}'); + // Only a read from the pool can return what the file does not say. + $keys = array_keys($this->pool->getValues()); + self::assertCount(1, $keys); + $item = $this->pool->getItem($keys[0]); + $this->pool->save($item->set(['messages' => ['app.title' => 'from the pool'], 'error' => null])); - self::assertSame(['app.title' => 'one'], $this->scanner()->getMessages('messages', 'fr')); - self::assertSame(['app.title' => 'two'], (new TranslationFileScanner($this->dir))->getMessages('messages', 'fr')); + self::assertSame(['app.title' => 'from the pool'], $this->scanner()->getMessages('messages', 'fr')); } - public function testAFileWhoseDateChangedIsParsedAgain(): void + /** A build that freezes the dates, a release deployed under the same path: same date, same size. */ + public function testAFileWhoseContentChangedIsParsedAgainWhateverItsDateAndSize(): void { $this->write('messages.fr.json', '{"app.title":"one"}'); + touch($this->dir.'/messages.fr.json', 1_760_000_000); $this->scanner()->getMessages('messages', 'fr'); - $this->write('messages.fr.json', '{"app.title":"two"}', self::MTIME + 1); + $this->write('messages.fr.json', '{"app.title":"two"}'); + touch($this->dir.'/messages.fr.json', 1_760_000_000); self::assertSame(['app.title' => 'two'], $this->scanner()->getMessages('messages', 'fr')); } - /** Two writes within the same second: the size tells them apart. */ - public function testAFileWhoseSizeChangedIsParsedAgain(): void + /** A chmod or a chown changes neither the content nor the date: the error is not cached. */ + public function testAFileMadeReadableAgainIsReadByTheNextProcess(): void { $this->write('messages.fr.json', '{"app.title":"one"}'); - $this->scanner()->getMessages('messages', 'fr'); + chmod($this->dir.'/messages.fr.json', 0o000); - $this->write('messages.fr.json', '{"app.title":"three"}'); + if (is_readable($this->dir.'/messages.fr.json')) { + self::markTestSkipped('The file stays readable (running as root).'); + } - self::assertSame(['app.title' => 'three'], $this->scanner()->getMessages('messages', 'fr')); + $scanner = $this->scanner(); + self::assertSame([], $scanner->getMessages('messages', 'fr')); + self::assertSame([$this->dir.'/messages.fr.json' => 'The file cannot be read: check its permissions.'], $scanner->getIgnoredFiles()); + + chmod($this->dir.'/messages.fr.json', 0o644); + $scanner = $this->scanner(); + + self::assertSame(['app.title' => 'one'], $scanner->getMessages('messages', 'fr')); + self::assertSame([], $scanner->getIgnoredFiles()); } public function testAnUnparseableFileIsStillReportedWhenServedFromTheCache(): void @@ -85,7 +98,7 @@ public function testAResetScannerSeesAnEditedFileAndANewOne(): void $scanner = $this->scanner(); $scanner->getMessages('messages', 'fr'); - $this->write('messages.fr.json', '{"app.title":"two"}', self::MTIME + 1); + $this->write('messages.fr.json', '{"app.title":"two"}'); $this->write('messages.de.json', '{"app.title":"eins"}'); self::assertSame(['app.title' => 'one'], $scanner->getMessages('messages', 'fr'), 'Memoized until the reset.'); @@ -100,9 +113,8 @@ private function scanner(): TranslationFileScanner return new TranslationFileScanner($this->dir, filesCache: $this->pool); } - private function write(string $name, string $content, int $mtime = self::MTIME): void + private function write(string $name, string $content): void { file_put_contents($this->dir.'/'.$name, $content); - touch($this->dir.'/'.$name, $mtime); } } From 7de45257fca05000cce0821d12465ac1076be1f7 Mon Sep 17 00:00:00 2001 From: Julien Dehais Date: Thu, 15 Oct 2026 14:38:14 +0200 Subject: [PATCH 5/5] docs: name parseFile() in the CACHE_FORMAT docblock, and say the loaders' version is not part of the key --- docs/adr/0027-parsed-files-cached-by-fingerprint.md | 4 +++- docs/operations/cache.md | 3 ++- src/Catalogue/TranslationFileScanner.php | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/adr/0027-parsed-files-cached-by-fingerprint.md b/docs/adr/0027-parsed-files-cached-by-fingerprint.md index 8bb5e61..1134dc5 100644 --- a/docs/adr/0027-parsed-files-cached-by-fingerprint.md +++ b/docs/adr/0027-parsed-files-cached-by-fingerprint.md @@ -36,7 +36,9 @@ is reset between two Messenger messages. - A request reads and hashes every file of the catalogues it reads instead of parsing them. - A shared pool needs nothing more, on several servers or between releases: an entry only matches the content it was parsed from. -- What the scanner stores changes only with a new format version in the key. +- What the scanner stores changes only with a new format version in the key. The version of + the loaders is not part of it: a parsing change in a new release of the translation + component is served once its entries expire, or the pool is emptied. ## Related diff --git a/docs/operations/cache.md b/docs/operations/cache.md index 7f05e51..e824d99 100644 --- a/docs/operations/cache.md +++ b/docs/operations/cache.md @@ -55,7 +55,8 @@ default). | Not cached | The walk of the directories: a file added or removed is seen on the next request. A file that cannot be read: once readable again, it is read on the next request | | Long-running processes | The scanner is reset between two Messenger messages (`kernel.reset`): a worker sees an edited file without a restart | -To empty the pool: +A new release of the translation component that parses a file differently is only seen once +the entry expires: empty the pool with `bin/console cache:pool:clear cyllene_digital_ai_translation.translation_files_cache`. `cache:clear` is not enough when `APP_SHARE_DIR` is set, as the Symfony 7.4 and 8 skeletons do (`var/share`): the filesystem `cache.app`, and this pool with it, then lives in that directory, diff --git a/src/Catalogue/TranslationFileScanner.php b/src/Catalogue/TranslationFileScanner.php index b4196ef..d946722 100644 --- a/src/Catalogue/TranslationFileScanner.php +++ b/src/Catalogue/TranslationFileScanner.php @@ -38,7 +38,7 @@ final class TranslationFileScanner implements ResetInterface private const string INTL_ICU_SUFFIX = '+intl-icu'; - /** Part of every persistent cache key: bump it when what loadFile() returns changes. */ + /** Part of every persistent cache key: bump it when what parseFile() returns changes. */ private const int CACHE_FORMAT = 1; /** @var array|null identifier => catalogue, sorted */