Implement PPL evaluator for qwen 3 0.6B

[zhenchaoni]

Add teacher-forced perplexity evaluation for genai causal LMs

Summary

Adds a text-generation eval task to winml eval that computes teacher-forced, disjoint fixed-length perplexity (PPL) for autoregressive causal LMs. The same evaluator scores both onnxruntime-genai bundles (ONNX, on-device) and PyTorch fp32 baselines through a single model-agnostic contract, so ONNX-vs-baseline accuracy regression can be tracked in the existing e2e harness.

Validated on Qwen/Qwen3-0.6B against wikitext-2-raw-v1 (test split):

• ONNX genai bundle PPL = 25.20
• PyTorch fp32 baseline PPL = 23.65

What's included

Evaluator (eval/text_generation_evaluator.py)

• WinMLTextGenerationEvaluator: model-agnostic, zero-branch. Builds a token corpus, splits it into non-overlapping seqlen blocks, and computes PPL via a numerically stable log-sum-exp NLL accumulator.
• Relies on a uniform scoring contract — encode(text) -> list[int] and forward(ids) -> out where out.logits is (1, len(ids)-1, vocab) with row i predicting token i+1. No per-model conditionals.

Model layer (models/winml/genai_causal_lm.py)

• WinMLGenaiCausalLM: causal-LM inference over an onnxruntime-genai bundle directory (loaded as-is, compile=False — building/compiling is winml build's job).
• HFCausalLM: PyTorch fp32 adapter implementing the same contract (lazy torch/transformers import, add_special_tokens=False to reproduce POC tokenization). Lives in src so the baseline script and the evaluator share one code path.
• CausalLMOutput shared output dataclass.

Wiring

• eval/evaluate.py: registers the text-generation evaluator, a default wikitext-2-raw-v1 dataset, and _load_genai_causal_lm (validates the bundle dir has genai_config.json + .onnx files before loading).
• commands/eval.py: _resolve_model_path now routes an existing local directory in -m to model_path (so a genai bundle dir is read from disk instead of being treated as a Hub id).
• utils/eval_utils.py: _TEXT_GENERATION_SCHEMA with input_column (default text), num_tokens (default 8192), seqlen (default 2048).

Baseline / accuracy harness

• run_pytorch_baseline.py: text-generation branch uses the shared HFCausalLM adapter.
• accuracy.py: adds a perplexity compare strategy (delta_relative, warn 0.05 / fail 0.10, lower-is-better).
• testsets/models_with_acc.json + cache/baseline_cache.json: Qwen/Qwen3-0.6B entry with cached fp32 baseline (PPL 23.645425).

Tests

• test_text_generation_evaluator.py (14): block-chunking protocol, corpus loading, uniform-logits → PPL = vocab-size sanity, NLL cross-entropy reference.
• test_genai_causal_lm.py (9): CausalLMOutput, encode/forward contract, float32 conversion, last-row trimming.

[DingmaomaoBJTU]

Please see the inline comments.

[DingmaomaoBJTU]

A few additional issues found while reviewing the updated changes.

[DingmaomaoBJTU]

LGTM. Well-structured text-generation perplexity evaluator. Key observations (non-blocking):

1. Teacher forcing via set_logits is clever — the one-token-at-a-time replay with forced logits correctly scores all positions while reusing the genai generator's KV cache. The neutralization of bundle search constraints (min_length=0,
   o_repeat_ngram_size=0, etc.) is essential and properly tested.

2. _step_nll is numerically sound — the log-sum-exp with max subtraction avoids overflow, and the f64 upcast avoids precision issues on large vocabs.

3. Default dataset column merge fix is a nice UX improvement — preserving user --column values while filling in defaults means --column num_tokens=1024 --column seqlen=512 works without needing to repeat --dataset.

4. Minor: _load_corpus_tokens uses model.encode(text) + 2 per row as an approximation, which could over-shoot significantly for short rows (many BPE boundary tokens). Fine for the first iteration, but a comment noting this is conservative (overcounts) would help.

5. Minor: HFCausalLM.forward computes the full sequence in one shot (model(input_ids=...)) which works for short seqlens but could OOM on very long sequences on GPU. Since the default seqlen is 2048 and this is a baseline path, acceptable — just noting for awareness.