fix: Salt failure is a hard error

[arekay-nv]

What does this PR do?

Makes salt failures a hard error instead of a silent skip. Previously, when --warmup-salt was on (the default) but a sample couldn't be salted, Dataset._apply_salt logged a warning (or silently passed the sample through) and the warmup ran with no cache-busting — the exact condition salt exists to prevent, failing quietly.

Now salt=True requires every sample to be a dict with a text (str) prompt and no input_tokens; anything else raises DatasetValidationError at dataset-load time, before any load is issued.

Key changes:

Area	Change
Validation	New Dataset.validate_saltable() — rejects non-dict samples, input_tokens (pre-tokenized; adapters send these verbatim so a salted prompt never reaches the server), missing prompt, and non-str prompt. Error names the offending sample index + remediation.
Fail-fast placement	Called from _load_datasets when warmup.enabled and warmup.salt, so an unsaltable dataset fails before worker/aggregator subprocesses spawn. Also called from with_salt() so the salting mechanism stays self-protecting.
Hot path	_apply_salt collapses to a single dict-merge ({**data, "prompt": f"[{salt}] {data['prompt']}"}) — the multimodal-list handling, input_tokens warnings, and silent passthroughs are deleted; the contract is guaranteed upstream.

⚠️ Behavior change (note for reviewers)

• Multimodal / pre-tokenized warmup + salt now hard-errors. A prompt that is a list (image/video workloads) or a dataset with input_tokens (e.g. gpt-oss-120b, DeepSeek-R1 via /v1/completions) will fail warmup instead of silently skipping salt. Fix: set --warmup-salt=false / warmup.salt: false.
• Whole-dataset validation. The run is rejected if any sample is non-conforming — including samples the n_requests subset would never issue. Stricter than the old per-sample skip; intended by the hard-error design.

Type of change

• Bug fix
• New feature
• Documentation update
• Refactor/cleanup

Related issues

Testing

• Tests added/updated
• All tests pass locally
• Manual testing completed

Checklist

• Code follows project style
• Pre-commit hooks pass
• Documentation updated (if needed)

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Code Review

This pull request introduces early validation for dataset salting during the benchmark setup phase. It adds a validate_saltable method to ensure that all dataset samples are compatible with salting (i.e., they are dictionaries containing a string prompt and no input_tokens) before any subprocesses are spawned. The review feedback highlights a critical bug in validate_saltable where iterating over self.data directly will fail if it is a pandas.DataFrame (as it would iterate over column names instead of rows), and provides a code suggestion to handle this case.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.