diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d44dd02..079ed5dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,12 @@ All notable changes to ForgeLM are documented here. ### Fixed +- `distributed.fsdp_strategy: no_shard` and `hybrid_shard` are rejected when the config loads (exit 1). They were documented, passed `--dry-run`, + and then failed with exit 2 at trainer construction (`fsdp=hybrid_shard is only available with FSDP1`), because transformers 5.10+ runs + FSDP2 by default. `fsdp_auto_wrap: false` and `fsdp_backward_prefetch: backward_post` are still accepted but change nothing under FSDP2; + ForgeLM now logs a warning when either is set, and the field descriptions, reference pages, user manuals and `config_template.yaml` + say so. Moving to FSDP2 settings (`fsdp=True` plus an explicit `fsdp_config`) is not part of this change; the string form is + deprecated in transformers and slated for removal in v5.20 (#104). - Fix the KTO bullet on the Alignment overview user-manual page (English and Turkish): it named the fields `(prompt, response, good?)`, but the KTO loader reads `prompt`, `completion` and a boolean `label` (#965). diff --git a/config_template.yaml b/config_template.yaml index 8aec1398..84fd1668 100644 --- a/config_template.yaml +++ b/config_template.yaml @@ -167,7 +167,7 @@ data: # strategy: "deepspeed" # "deepspeed" or "fsdp" # deepspeed_config: "zero2" # preset: "zero2", "zero3", "zero3_offload" or path to JSON # # --- FSDP options (when strategy: "fsdp") --- -# # fsdp_strategy: "full_shard" # "full_shard", "shard_grad_op", "no_shard", "hybrid_shard" +# # fsdp_strategy: "full_shard" # "full_shard" or "shard_grad_op" (FSDP2 supports no other) # # fsdp_auto_wrap: true # # fsdp_offload: false diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index b78fa07f..c74af168 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -398,10 +398,10 @@ kaynağı kaydedilmemiştir. |------|-----|-----------|----------| | `strategy` | string | `null` | `"deepspeed"` veya `"fsdp"` (null = tek GPU) | | `deepspeed_config` | string | `null` | Ön ayar (`"zero2"`, `"zero3"`, `"zero3_offload"`) veya JSON yolu | -| `fsdp_strategy` | string | `"full_shard"` | `"full_shard"`, `"shard_grad_op"`, `"hybrid_shard"`, `"no_shard"` | -| `fsdp_auto_wrap` | bool | `true` | Transformer katmanlarını otomatik sar | +| `fsdp_strategy` | string | `"full_shard"` | `"full_shard"` veya `"shard_grad_op"` (`"hybrid_shard"` ve `"no_shard"` reddedilir: FSDP2 bunları desteklemez) | +| `fsdp_auto_wrap` | bool | `true` | Transformer katmanlarını otomatik sar (transformers 5.10+'ta yok sayılır, katmanlar hep sarılır) | | `fsdp_offload` | bool | `false` | Parametreleri CPU'ya taşı | -| `fsdp_backward_prefetch` | string | `"backward_pre"` | `"backward_pre"` veya `"backward_post"` | +| `fsdp_backward_prefetch` | string | `"backward_pre"` | `"backward_pre"` veya `"backward_post"` (transformers 5.10+'ta yok sayılır; FSDP2 prefetch politikası almaz) | | `fsdp_state_dict_type` | string | `"FULL_STATE_DICT"` | `"FULL_STATE_DICT"` veya `"SHARDED_STATE_DICT"` | ## `synthetic` (İsteğe bağlı — Sentetik Veri Üretimi) diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 0f83d16e..ae9e907c 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -310,10 +310,10 @@ silently extend the retention horizon by re-using a stale workspace. |-------|------|---------|-------------| | `strategy` | string | `null` | `"deepspeed"` or `"fsdp"` (null = single GPU) | | `deepspeed_config` | string | `null` | Preset (`"zero2"`, `"zero3"`, `"zero3_offload"`) or JSON path | -| `fsdp_strategy` | string | `"full_shard"` | `"full_shard"`, `"shard_grad_op"`, `"hybrid_shard"`, `"no_shard"` | -| `fsdp_auto_wrap` | bool | `true` | Auto-wrap transformer layers | +| `fsdp_strategy` | string | `"full_shard"` | `"full_shard"` or `"shard_grad_op"` (`"hybrid_shard"` and `"no_shard"` are rejected: FSDP2 does not support them) | +| `fsdp_auto_wrap` | bool | `true` | Auto-wrap transformer layers (ignored on transformers 5.10+, which always wraps them) | | `fsdp_offload` | bool | `false` | Offload parameters to CPU | -| `fsdp_backward_prefetch` | string | `"backward_pre"` | `"backward_pre"` or `"backward_post"` | +| `fsdp_backward_prefetch` | string | `"backward_pre"` | `"backward_pre"` or `"backward_post"` (ignored on transformers 5.10+, whose FSDP2 takes no prefetch policy) | | `fsdp_state_dict_type` | string | `"FULL_STATE_DICT"` | `"FULL_STATE_DICT"` or `"SHARDED_STATE_DICT"` | --- diff --git a/docs/reference/distributed_training-tr.md b/docs/reference/distributed_training-tr.md index 32e6378f..0df37e60 100644 --- a/docs/reference/distributed_training-tr.md +++ b/docs/reference/distributed_training-tr.md @@ -120,8 +120,8 @@ PyTorch Fully Sharded Data Parallel (FSDP), DeepSpeed'e alternatif olarak PyTorc |----------|----------|-----------------| | `full_shard` | Parametreleri, gradyanları ve optimizer durumlarını böl | En Yüksek | | `shard_grad_op` | Yalnızca gradyanları ve optimizer durumlarını böl | Orta | -| `hybrid_shard` | Düğüm içi tam bölme, düğümler arası çoğaltma | Çoklu Düğüm | -| `no_shard` | Standart DDP (bölme yok) | Yok | + +`hybrid_shard` ve `no_shard` **kabul edilmez**: `fsdp_strategy` `full_shard` ya da `shard_grad_op` olmalıdır. ForgeLM'in gerektirdiği transformers 5.10 ve sonrası varsayılan olarak FSDP2 çalıştırır ve FSDP2 bu ikisini de desteklemez: eskiden `--dry-run`'ı geçip model yüklendikten sonra exit 2 ile (`fsdp=hybrid_shard is only available with FSDP1`) patlıyorlardı, şimdi doğrulamada exit 1 ile reddediliyorlar. ### Kullanım @@ -129,12 +129,14 @@ PyTorch Fully Sharded Data Parallel (FSDP), DeepSpeed'e alternatif olarak PyTorc distributed: strategy: "fsdp" fsdp_strategy: "full_shard" - fsdp_auto_wrap: true # Auto-wrap transformer layers + fsdp_auto_wrap: true # transformers 5.10+'ta yok sayılır (FSDP2 transformer katmanlarını hep sarar) fsdp_offload: false # Offload parameters AND gradients (between forward/backward) to CPU - fsdp_backward_prefetch: "backward_pre" # Prefetch strategy + fsdp_backward_prefetch: "backward_pre" # transformers 5.10+'ta yok sayılır (FSDP2 prefetch politikası almaz) fsdp_state_dict_type: "FULL_STATE_DICT" # State dict handling ``` +`fsdp_auto_wrap: false` ve `fsdp_backward_prefetch: backward_post` hâlâ kabul edilir, ancak FSDP2'de hiçbir şeyi değiştirmezler; bu yüzden ikisinden biri varsayılanından farklı ayarlandığında ForgeLM bir uyarı loglar. + ### FSDP'yi DeepSpeed yerine ne zaman seçmeli - **FSDP**: Native PyTorch, ekstra bağımlılık yok, daha basit kurulum, diff --git a/docs/reference/distributed_training.md b/docs/reference/distributed_training.md index b83b29f9..013018a0 100644 --- a/docs/reference/distributed_training.md +++ b/docs/reference/distributed_training.md @@ -118,8 +118,8 @@ PyTorch Fully Sharded Data Parallel (FSDP) is an alternative to DeepSpeed that's |----------|-------------|----------------| | `full_shard` | Shard parameters, gradients, and optimizer states | Highest | | `shard_grad_op` | Shard gradients and optimizer states only | Medium | -| `hybrid_shard` | Full shard within nodes, replicate across nodes | Multi-node | -| `no_shard` | Standard DDP (no sharding) | None | + +`hybrid_shard` and `no_shard` are **not accepted**: `fsdp_strategy` must be `full_shard` or `shard_grad_op`. transformers 5.10 and later, which ForgeLM requires, run FSDP2 by default, and FSDP2 supports neither: they used to pass `--dry-run` and then fail with exit 2 (`fsdp=hybrid_shard is only available with FSDP1`) after the model was loaded, and now fail validation with exit 1. ### Usage @@ -127,12 +127,14 @@ PyTorch Fully Sharded Data Parallel (FSDP) is an alternative to DeepSpeed that's distributed: strategy: "fsdp" fsdp_strategy: "full_shard" - fsdp_auto_wrap: true # Auto-wrap transformer layers + fsdp_auto_wrap: true # Ignored on transformers 5.10+ (FSDP2 always wraps transformer layers) fsdp_offload: false # Offload parameters AND gradients (between forward/backward) to CPU - fsdp_backward_prefetch: "backward_pre" # Prefetch strategy + fsdp_backward_prefetch: "backward_pre" # Ignored on transformers 5.10+ (FSDP2 takes no prefetch policy) fsdp_state_dict_type: "FULL_STATE_DICT" # State dict handling ``` +`fsdp_auto_wrap: false` and `fsdp_backward_prefetch: backward_post` are still accepted, but under FSDP2 they change nothing, so ForgeLM logs a warning when either is set away from its default. + ### When to Choose FSDP over DeepSpeed - **FSDP**: Native PyTorch, no extra dependency, simpler setup, good for most use cases diff --git a/docs/usermanuals/en/reference/configuration.md b/docs/usermanuals/en/reference/configuration.md index c4ba73c2..42ebf4cb 100644 --- a/docs/usermanuals/en/reference/configuration.md +++ b/docs/usermanuals/en/reference/configuration.md @@ -229,10 +229,10 @@ The merge field is `method` (not `algorithm`), and there is no `dare_ties` choic distributed: strategy: null # null (single-GPU, no distributed wrapping) | deepspeed | fsdp deepspeed_config: null # path to a DeepSpeed JSON, or preset name: zero2 | zero3 | zero3_offload - fsdp_strategy: "full_shard" # full_shard | shard_grad_op | no_shard | hybrid_shard - fsdp_auto_wrap: true # auto-wrap transformer layers (recommended) + fsdp_strategy: "full_shard" # full_shard | shard_grad_op + fsdp_auto_wrap: true # ignored on transformers 5.10+ (FSDP2 always wraps transformer layers) fsdp_offload: false # offload FSDP parameters to CPU between forward/backward - fsdp_backward_prefetch: "backward_pre" # backward_pre | backward_post + fsdp_backward_prefetch: "backward_pre" # backward_pre | backward_post; ignored on transformers 5.10+ fsdp_state_dict_type: "FULL_STATE_DICT" # FULL_STATE_DICT | SHARDED_STATE_DICT ``` diff --git a/docs/usermanuals/en/training/distributed.md b/docs/usermanuals/en/training/distributed.md index 7aa622ff..8679d546 100644 --- a/docs/usermanuals/en/training/distributed.md +++ b/docs/usermanuals/en/training/distributed.md @@ -111,12 +111,14 @@ FSDP shards similarly to ZeRO-3 but uses PyTorch's native FullyShardedDataParall ```yaml distributed: strategy: "fsdp" - fsdp_strategy: "full_shard" # full_shard | shard_grad_op | no_shard | hybrid_shard - fsdp_auto_wrap: true # auto-wrap transformer layers (recommended) + fsdp_strategy: "full_shard" # full_shard | shard_grad_op + fsdp_auto_wrap: true # ignored on transformers 5.10+ (FSDP2 always wraps transformer layers) fsdp_offload: false # offload parameters to CPU between forward/backward fsdp_state_dict_type: "FULL_STATE_DICT" # FULL_STATE_DICT | SHARDED_STATE_DICT ``` +`no_shard` and `hybrid_shard` are rejected when the config loads: transformers 5.10+, which ForgeLM requires, runs FSDP2, and FSDP2 supports neither. `fsdp_auto_wrap: false` and `fsdp_backward_prefetch: backward_post` are still accepted but change nothing under FSDP2, so ForgeLM logs a warning when either is set. + `DistributedConfig` has no `fsdp_auto_wrap_policy` or `fsdp_offload_params` field — auto-wrap is the plain boolean `fsdp_auto_wrap`, and CPU offload is `fsdp_offload` (no `_params` suffix). ## Gradient accumulation diff --git a/docs/usermanuals/tr/reference/configuration.md b/docs/usermanuals/tr/reference/configuration.md index 39f8e44b..9620007e 100644 --- a/docs/usermanuals/tr/reference/configuration.md +++ b/docs/usermanuals/tr/reference/configuration.md @@ -231,10 +231,10 @@ Merge alanı `algorithm` değil `method`'dur, `dare_ties` seçeneği, `base_mode distributed: strategy: null # null (tek GPU, dağıtık sarmalama yok) | deepspeed | fsdp deepspeed_config: null # bir DeepSpeed JSON'a yol, veya preset adı: zero2 | zero3 | zero3_offload - fsdp_strategy: "full_shard" # full_shard | shard_grad_op | no_shard | hybrid_shard - fsdp_auto_wrap: true # transformer katmanlarını otomatik sarmala (önerilir) + fsdp_strategy: "full_shard" # full_shard | shard_grad_op + fsdp_auto_wrap: true # transformers 5.10+'ta yok sayılır (FSDP2 transformer katmanlarını hep sarar) fsdp_offload: false # forward/backward arasında FSDP parametrelerini CPU'ya offload et - fsdp_backward_prefetch: "backward_pre" # backward_pre | backward_post + fsdp_backward_prefetch: "backward_pre" # backward_pre | backward_post; transformers 5.10+'ta yok sayılır fsdp_state_dict_type: "FULL_STATE_DICT" # FULL_STATE_DICT | SHARDED_STATE_DICT ``` diff --git a/docs/usermanuals/tr/training/distributed.md b/docs/usermanuals/tr/training/distributed.md index 4dcfd239..7e73a6d7 100644 --- a/docs/usermanuals/tr/training/distributed.md +++ b/docs/usermanuals/tr/training/distributed.md @@ -111,12 +111,14 @@ FSDP, ZeRO-3 gibi sharder ama PyTorch'un yerli FullyShardedDataParallel'ini kull ```yaml distributed: strategy: "fsdp" - fsdp_strategy: "full_shard" # full_shard | shard_grad_op | no_shard | hybrid_shard - fsdp_auto_wrap: true # transformer katmanlarını otomatik sar (önerilir) + fsdp_strategy: "full_shard" # full_shard | shard_grad_op + fsdp_auto_wrap: true # transformers 5.10+'ta yok sayılır (FSDP2 transformer katmanlarını hep sarar) fsdp_offload: false # forward/backward arasında parametreleri CPU'ya boşalt fsdp_state_dict_type: "FULL_STATE_DICT" # FULL_STATE_DICT | SHARDED_STATE_DICT ``` +`no_shard` ve `hybrid_shard` config yüklenirken reddedilir: ForgeLM'in gerektirdiği transformers 5.10+ FSDP2 çalıştırır ve FSDP2 bu ikisini desteklemez. `fsdp_auto_wrap: false` ve `fsdp_backward_prefetch: backward_post` hâlâ kabul edilir ancak FSDP2'de hiçbir şeyi değiştirmez; bu yüzden ikisinden biri ayarlandığında ForgeLM bir uyarı loglar. + `DistributedConfig`'te `fsdp_auto_wrap_policy` veya `fsdp_offload_params` alanı yoktur — auto-wrap düz bir `fsdp_auto_wrap` boolean'ıdır, CPU offload ise `fsdp_offload`'dur (`_params` eki yoktur). ## Gradient accumulation diff --git a/forgelm/cli/_config_load.py b/forgelm/cli/_config_load.py index 4e5ac477..8bc890f8 100644 --- a/forgelm/cli/_config_load.py +++ b/forgelm/cli/_config_load.py @@ -55,3 +55,27 @@ def _apply_offline_flag(config, offline_arg: bool) -> None: os.environ["TRANSFORMERS_OFFLINE"] = "1" os.environ["HF_DATASETS_OFFLINE"] = "1" logger.info("Offline mode enabled. All HF Hub network calls are disabled.") + + +def _warn_inert_fsdp_options(config) -> None: + """Say so when an FSDP knob that transformers' FSDP2 default ignores is set away from its default. + + ``fsdp_auto_wrap: false`` and ``fsdp_backward_prefetch: backward_post`` are still accepted, but under + transformers 5.10+ (FSDP2) the transformer-layer wrap policy is always applied and no backward prefetch is + passed on, so the run behaves as if the defaults had been kept. (This lives here, not in the schema, to keep + ``config.py`` within its deferred-split line budget.) + """ + dist = getattr(config, "distributed", None) + if getattr(dist, "strategy", None) != "fsdp": + return + if not dist.fsdp_auto_wrap: + logger.warning( + "distributed.fsdp_auto_wrap=false has no effect: transformers 5.10+ runs FSDP2, which always wraps " + "transformer layers." + ) + if dist.fsdp_backward_prefetch != "backward_pre": + logger.warning( + "distributed.fsdp_backward_prefetch=%r has no effect: transformers 5.10+ runs FSDP2, which does not take " + "a backward-prefetch policy.", + dist.fsdp_backward_prefetch, + ) diff --git a/forgelm/cli/_dispatch.py b/forgelm/cli/_dispatch.py index e4b64387..62cdbc78 100644 --- a/forgelm/cli/_dispatch.py +++ b/forgelm/cli/_dispatch.py @@ -10,7 +10,7 @@ import sys from .._strict_json import dumps_strict -from ._config_load import _apply_offline_flag, _load_config_or_exit +from ._config_load import _apply_offline_flag, _load_config_or_exit, _warn_inert_fsdp_options from ._exit_codes import EXIT_CONFIG_ERROR, EXIT_SUCCESS, EXIT_TRAINING_ERROR, _clamp_exit_code from ._logging import _setup_logging, logger from ._no_train_modes import _maybe_run_no_train_mode @@ -241,6 +241,7 @@ def _main_inner() -> None: config = _load_config_or_exit(args.config, json_output) _apply_offline_flag(config, args.offline) + _warn_inert_fsdp_options(config) if config.pipeline is not None: _dispatch_pipeline_mode(config, args, json_output) diff --git a/forgelm/config.py b/forgelm/config.py index 4114f2df..b13d3daf 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -770,17 +770,17 @@ class DistributedConfig(BaseModel): default=None, description="DeepSpeed config: filesystem path to a DS JSON OR preset name (`zero2`, `zero3`, `zero3_offload`).", ) - fsdp_strategy: Literal["full_shard", "shard_grad_op", "no_shard", "hybrid_shard"] = Field( + fsdp_strategy: Literal["full_shard", "shard_grad_op"] = Field( default="full_shard", - description="FSDP sharding strategy. `full_shard` is the production default; `hybrid_shard` for multi-node intra-node sharding.", + description="FSDP sharding strategy. `full_shard` is the production default; `shard_grad_op` shards gradients and optimizer states only. `no_shard` and `hybrid_shard` are not accepted: transformers 5.10+ runs FSDP2, which supports neither.", ) - fsdp_auto_wrap: bool = Field(default=True, description="FSDP: auto-wrap transformer layers (recommended).") + fsdp_auto_wrap: bool = Field(default=True, description="FSDP auto-wrap; ignored by FSDP2 (transformers 5.10+).") fsdp_offload: bool = Field( default=False, description="FSDP: offload parameters to CPU between forward and backward (slower, less VRAM)." ) fsdp_backward_prefetch: Literal["backward_pre", "backward_post"] = Field( default="backward_pre", - description="FSDP backward-prefetch policy. `backward_pre` overlaps comm + compute; `backward_post` is more memory-conservative.", + description="FSDP backward-prefetch policy. Not honoured by transformers 5.10+ (FSDP2 prefetches on its own); `backward_post` logs a warning.", ) fsdp_state_dict_type: Literal["FULL_STATE_DICT", "SHARDED_STATE_DICT"] = Field( default="FULL_STATE_DICT", diff --git a/tests/test_distributed.py b/tests/test_distributed.py index d7127735..b51d7819 100644 --- a/tests/test_distributed.py +++ b/tests/test_distributed.py @@ -66,9 +66,9 @@ def test_deepspeed_config(self, minimal_config): assert cfg.distributed.deepspeed_config == "zero3" def test_fsdp_config(self, minimal_config): - cfg = ForgeConfig(**minimal_config(distributed={"strategy": "fsdp", "fsdp_strategy": "hybrid_shard"})) + cfg = ForgeConfig(**minimal_config(distributed={"strategy": "fsdp", "fsdp_strategy": "shard_grad_op"})) assert cfg.distributed.strategy == "fsdp" - assert cfg.distributed.fsdp_strategy == "hybrid_shard" + assert cfg.distributed.fsdp_strategy == "shard_grad_op" def test_unsloth_distributed_raises(self, minimal_config): """Unsloth + distributed should raise ValueError.""" @@ -306,3 +306,113 @@ def test_fsdp_strategy_does_not_require_deepspeed(self, monkeypatch, minimal_con kwargs: dict = {} ForgeTrainer._apply_distributed_config(object.__new__(ForgeTrainer), kwargs, cfg.distributed) assert "deepspeed" not in kwargs + + +# --- FSDP: strategies transformers' FSDP2 default cannot express (#104) --- + + +class TestFsdp1OnlyStrategiesAreRejectedAtConfigTime: + """``no_shard`` and ``hybrid_shard`` used to pass ``--dry-run`` and exit 2 at trainer construction. + + transformers 5.10+ runs FSDP2 by default and raises ``fsdp=hybrid_shard is only available with FSDP1`` when + ``TrainingArguments`` is built, so the config is refused up front (exit 1) instead of after the model is loaded. + """ + + @pytest.mark.parametrize("strategy", ["no_shard", "hybrid_shard"]) + def test_the_schema_refuses_them(self, strategy): + with pytest.raises(ValidationError, match="fsdp_strategy"): + DistributedConfig(strategy="fsdp", fsdp_strategy=strategy) + + @pytest.mark.parametrize("strategy", ["no_shard", "hybrid_shard"]) + def test_dry_run_exits_with_a_config_error(self, tmp_path, minimal_config, strategy): + from unittest.mock import patch + + from forgelm.cli import EXIT_CONFIG_ERROR, main + + cfg_path = tmp_path / "config.yaml" + cfg_path.write_text(yaml.dump(minimal_config(distributed={"strategy": "fsdp", "fsdp_strategy": strategy}))) + with patch("sys.argv", ["forgelm", "--config", str(cfg_path), "--dry-run"]): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == EXIT_CONFIG_ERROR + + @pytest.mark.parametrize("strategy", ["full_shard", "shard_grad_op"]) + @pytest.mark.parametrize("auto_wrap", [True, False]) + @pytest.mark.parametrize("offload", [True, False]) + @pytest.mark.parametrize("prefetch", ["backward_pre", "backward_post"]) + @pytest.mark.parametrize("state_dict", ["FULL_STATE_DICT", "SHARDED_STATE_DICT"]) + def test_every_accepted_variant_builds_training_arguments( + self, tmp_path, strategy, auto_wrap, offload, prefetch, state_dict + ): + """What the schema accepts must be what ``TrainingArguments`` accepts (no config that dies later).""" + from types import SimpleNamespace + + from transformers import TrainingArguments + + from forgelm.trainer import ForgeTrainer + + dist_cfg = DistributedConfig( + strategy="fsdp", + fsdp_strategy=strategy, + fsdp_auto_wrap=auto_wrap, + fsdp_offload=offload, + fsdp_backward_prefetch=prefetch, + fsdp_state_dict_type=state_dict, + ) + kwargs: dict = {} + ForgeTrainer._apply_distributed_config(SimpleNamespace(), kwargs, dist_cfg) + TrainingArguments(output_dir=str(tmp_path), use_cpu=True, report_to=[], **kwargs) + + +class TestInertFsdpOptionsAreCalledOut: + """``fsdp_auto_wrap: false`` and ``fsdp_backward_prefetch: backward_post`` are ignored by FSDP2; say so (#104).""" + + @staticmethod + def _config(**distributed): + from tests._helpers.factories import minimal_config + + return ForgeConfig(**minimal_config(distributed={"strategy": "fsdp", **distributed})) + + @staticmethod + def _warnings(config, caplog): + import logging + + from forgelm.cli._config_load import _warn_inert_fsdp_options + + with caplog.at_level(logging.WARNING): + _warn_inert_fsdp_options(config) + return [r.getMessage() for r in caplog.records if "distributed.fsdp_" in r.getMessage()] + + def test_the_defaults_are_silent(self, caplog): + assert self._warnings(self._config(), caplog) == [] + + def test_turning_auto_wrap_off_is_called_out(self, caplog): + warned = self._warnings(self._config(fsdp_auto_wrap=False), caplog) + assert len(warned) == 1 and "fsdp_auto_wrap=false has no effect" in warned[0] + + def test_backward_post_is_called_out(self, caplog): + warned = self._warnings(self._config(fsdp_backward_prefetch="backward_post"), caplog) + assert len(warned) == 1 and "backward_post" in warned[0] + + def test_other_strategies_and_no_distributed_block_are_silent(self, caplog): + from tests._helpers.factories import minimal_config + + deepspeed = ForgeConfig(**minimal_config(distributed={"strategy": "deepspeed", "fsdp_auto_wrap": False})) + plain = ForgeConfig(**minimal_config()) + assert self._warnings(deepspeed, caplog) == [] + assert self._warnings(plain, caplog) == [] + + def test_a_dry_run_reaches_the_warning(self, tmp_path, caplog, minimal_config): + import logging + from unittest.mock import patch + + from forgelm.cli import EXIT_SUCCESS, main + + cfg_path = tmp_path / "config.yaml" + cfg_path.write_text(yaml.dump(minimal_config(distributed={"strategy": "fsdp", "fsdp_auto_wrap": False}))) + with caplog.at_level(logging.WARNING): + with patch("sys.argv", ["forgelm", "--config", str(cfg_path), "--dry-run"]): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == EXIT_SUCCESS + assert any("fsdp_auto_wrap=false has no effect" in r.getMessage() for r in caplog.records)