smaLLMs

CLI-first local LLM benchmarking with supported benchmark suites, live terminal progress, and structured artifacts for local leaderboards.

This repo is opinionated about scope:

• it is built for small local language models
• supported benchmarks are limited to tasks and context bands that make sense for those models
• frontier-scale context bands that do not fit realistic small-model local runs are intentionally not supported

smaLLMs is built for:

• local models running through Ollama or LM Studio
• cross-platform terminal use on macOS, Linux, Windows 11, and WSL
• reproducible benchmark runs with per-sample artifacts
• automatic benchmark dataset downloads with local caching outside the repo
• explicit offline runs after the benchmark cache has been warmed

Important distinction:

• model inference is local-first through Ollama or LM Studio
• benchmark datasets download automatically through the Hugging Face datasets ecosystem on first use
• after that, smaLLMs reuses the local cache automatically
• benchmark rows are cached outside the repo in a per-user cache directory, so the git repo itself stays small
• generated benchmark artifacts and website export bundles are local outputs and stay gitignored by default
• exported metadata avoids hostnames and other user-specific absolute path details

The goal is not to replace broad hosted systems like HELM or lm-eval-harness. smaLLMs is deliberately focused on local, open-weight model evaluation where the run should be inspectable from one machine: local model inventory, warmed dataset cache, redacted config snapshot, git revision, system metadata, sample-level JSONL, and website-ready exports.

What smaLLMs is trying to be

Not a toy wrapper around a few prompts.

smaLLMs is meant to become a serious open benchmarking CLI for local models. That means:

• the default interface is the terminal, not a web app
• the interactive mode uses arrow keys and multi-select menus
• benchmark runs emit machine-readable artifacts for downstream sites and leaderboards
• the CLI lists the benchmarks and suites that actually run today

Current runtime support

Supported local providers:

• Ollama
• LM Studio

Platform targets:

• macOS
• Linux
• Windows 11
• WSL with Windows-hosted Ollama fallback support

Supported Benchmarks

These benchmarks are supported by the local runner today:

• gsm8k
• mmlu
• mmlu_pro
• math
• aime_2024
• aime_2025
• arc_challenge
• arc_easy
• hellaswag
• winogrande
• boolq
• commonsense_qa
• piqa
• social_iqa
• openbookqa
• truthfulqa_mc1
• bbh_boolean_expressions
• graphwalks_bfs_0_128k
• graphwalks_parents_0_128k
• mrcr_v2_8needle_4k_8k
• mrcr_v2_8needle_8k_16k
• mrcr_v2_8needle_16k_32k
• mrcr_v2_8needle_32k_64k
• mrcr_v2_8needle_64k_128k

Quick start

1. Run the launcher

Windows 11:

py -3 start.py

If the Python launcher is not available, run the script with whichever Python 3.10+ interpreter you use locally.

or double-click:

.\start.bat

macOS / Linux / WSL:

python3 start.py

The launcher does the user-friendly path automatically:

• creates .venv if you are not already in a virtual environment
• installs the standard local runtime from requirements.txt
• checks Ollama and LM Studio status
• tells you whether existing Ollama models were already found
• opens the arrow-key terminal UI

Optional: install the CLI in editable mode

For development or repeated use:

python -m pip install -e .
smallms doctor
smallms benchmarks

The smallms console command maps to the same maintained CLI as python smaLLMs.py.

2. If you already have Ollama models installed

You do not need to pull them again.

smaLLMs automatically reuses whatever ollama list already shows on your machine. The only thing you need is for Ollama itself to be running.

3. If you do not have a local model yet

The fastest path is:

ollama pull llama3.2

You can also use LM Studio instead; just load a model there and keep its local server enabled.

4. Use the arrow-key terminal UI

The default interface is terminal-native:

• arrow keys to move
• space to toggle multi-select items
• enter to confirm
• q or esc to go back

Dependency files

• requirements.txt is the standard local install for normal users.
• requirements-dev.txt is only for development work on the repo.

Testing

Use the root test runner as the canonical local and CI entrypoint:

python test_everything.py

It discovers every test under tests/ and is the same path used by GitHub Actions and the PortUI test action.

Non-interactive CLI

The advanced commands are still available, but they are optional now.

Discover local models:

python3 smaLLMs.py doctor
python3 smaLLMs.py discover

Inspect supported suites and benchmarks:

python3 smaLLMs.py benchmarks

Run the default core suite on every discovered local model:

python3 smaLLMs.py quick --samples 3

Run a coding-agent harness comparison:

python3 smaLLMs.py agent-harness --dry-run
python3 smaLLMs.py agent-harness --harnesses pi opencode codex --tasks median_bugfix cli_feature path_safety

The agent-harness runner compares external coding-agent CLIs on deterministic local code-edit fixtures. It currently knows about Pi, OpenCode, and Codex CLI, using GPT-5.5/xhigh defaults where those harnesses expose them. Dry runs prepare the fixtures and commands without calling model providers. Real runs write artifacts under artifacts/agent_harness/runs/<run_id>/. Current local findings are summarized in docs/agent-harness-findings.md.

Agent-harness artifacts include per-row timestamps, return codes, duration, changed files, patch hashes, log byte/line counts, best-effort peak RSS via GNU time -v, reported token usage when the harness prints it, and explicit null values for token/context/cost fields the harness does not report. The CLI also mirrors a compact public JSON to the sibling website by default:

• ../websmaLLMs/public/data/agent-harness/latest.json
• ../websmaLLMs/public/data/agent-harness/runs/<run_id>.json

Run a separate Codex reasoning-effort sweep. This is deliberately separate from both the normal Ollama/LM Studio session feed and the harness-comparison feed:

python3 run_local_benchmarks.py reasoning-sweep --tasks median_bugfix

The default matrix follows the current Codex catalog: Sol and Terra at low, medium, high, xhigh, max, and ultra; Luna at low, medium, high, xhigh, and max. Codex CLI does not expose a sampling-temperature control for this workflow, so the third feed is named reasoning-efforts and records sampling_temperature: null explicitly. The public export is:

• ../websmaLLMs/public/data/reasoning-efforts/latest.json
• ../websmaLLMs/public/data/reasoning-efforts/runs/<run_id>.json

Use --tasks median_bugfix for a lower-usage effort curve or omit it for the full three-task fixture comparison. Interrupted sweeps keep completed per-variant artifacts under artifacts/reasoning_sweep/; only completed variants should be promoted to a public latest feed.

Run a specific suite:

python3 smaLLMs.py run --models llama3.2 qwen2.5:0.5b --benchmarks frontier_report_suite --samples 10

Export the latest artifacts:

python3 smaLLMs.py export

PortUI

smaLLMs includes a project-local PortUI app in portui/ plus vendored launcher wrappers, so the repo can expose the same command surface on Windows, macOS, Linux, and WSL without a global install.

Linux or macOS:

sh ./portui.sh --list
sh ./portui.sh --run doctor

Windows:

.\portui.ps1 -List
.\portui.ps1 -Run doctor

Command Prompt:

portui.cmd --list
portui.cmd --run doctor

The bundled actions cover the interactive launcher, setup check, doctor, model discovery, benchmark listing, quick suite, quick-suite cache warming, offline quick suite, exports, sibling websmaLLMs sync, and the unit test suite. The manifest uses PortUI's built-in {{projectDir}}, {{workspaceDir}}, and path separator variables so the same action files work when the repo is moved or cloned elsewhere.

If you are developing the PortUI engine itself and have the sibling portui repo checked out next to smaLLMs, refresh the vendored runtime with:

..\portui\portui.ps1 -InstallProject .

Artifact outputs

Each run writes structured artifacts to:

• artifacts/runs/<run_id>/

Website-friendly bundles are exported to:

• website_exports/latest/
• website_exports/runs/<run_id>/

Useful files:

• manifest.json
• summary.json
• leaderboard.json
• leaderboard.csv
• session.json for the website
• per-model/per-benchmark sample JSONL artifacts

For Ollama runs, smaLLMs uses the chat API for normal benchmark requests and disables thinking where the model supports it, so short-answer tasks do not spend the entire output budget in hidden reasoning. GPT-OSS models cannot fully disable thinking in Ollama, so smaLLMs requests the low thinking level and raises the response budget enough for the final answer channel to appear. If a model still returns an empty normal response, the runner records a raw_fallback_attempted / used_raw_fallback flag in the sample metadata and tries one raw-mode rescue request. A high raw fallback rate means the run is still fighting model formatting or token-budget behavior, so treat it as an artifact quality signal.

Every run manifest includes reproducibility evidence:

• selected model inventory, including Ollama digests when the local API exposes them
• git commit, branch, and dirty-worktree state
• operating system, Python, memory, CPU, and Ollama version metadata when available
• redacted effective config snapshot plus a stable config SHA-256
• dataset cache readiness, cache row counts, and cache file SHA-256 hashes
• explicit execution policy showing whether remote dataset downloads were allowed
• prompt template IDs, prompt template SHA-256 hashes, prompt SHA-256 hashes, and stable sample IDs

Every run summary and benchmark result also reports:

• accuracy with Wilson 95% confidence intervals
• response rate and invalid-prediction rate
• valid/invalid parsed prediction counts separate from provider errors
• raw fallback rate as a model formatting and provider-path quality signal

Each run directory includes a human-readable RUN_CARD.md next to the JSON artifacts. The run card summarizes the result, confidence intervals, offline policy, git/config/system fingerprints, cache hashes, and leaderboard rows in one reviewable Markdown report.

See docs/METHODOLOGY.md for the evaluation methodology and run-card publishing workflow.

Offline workflow

Warm the dataset cache once while online:

python smaLLMs.py cache --benchmarks quick_suite --samples 25

Then run without remote dataset downloads:

python smaLLMs.py run --benchmarks quick_suite --samples 25 --offline

In offline mode, smaLLMs fails before model execution if any selected benchmark lacks enough cached rows. Local Ollama or LM Studio HTTP calls are still allowed because those are the model runtime.

Website workflow

The exporter now writes a single self-contained website bundle:

• website_exports/latest/session.json

That file includes:

• run metadata
• leaderboard rows
• per-benchmark evaluation summaries
• embedded sample records with prompts, responses, parsed answers, token counts, latency, and raw provider metadata

When the sibling repo exists at ../websmaLLMs, the exporter also mirrors the latest session into:

• ../websmaLLMs/public/data/latest-session.json
• ../websmaLLMs/public/data/agent-harness/latest.json for coding-agent harness runs

That means the website can either:

• auto-load the mirrored session on startup
• auto-load the latest mirrored agent-harness run in its harness panel
• import any exported session.json manually through the UI

If you want to override the mirror location, use:

python3 run_local_benchmarks.py export --sync-dir /path/to/websmaLLMs/public/data
python3 smaLLMs.py agent-harness --sync-dir /path/to/websmaLLMs/public/data

Benchmark suites

The suite catalog follows the same rule as the benchmark list: higher-context tasks are allowed only up to bands that are still realistic for small local models. Frontier-scale 256K-1M character bands are intentionally excluded.

Runnable suites currently include:

• quick_suite
• core_suite
• knowledge_suite
• commonsense_suite
• reasoning_suite
• frontier_report_suite
• serious_suite
• all_benchmarks

Project principles

• Local-first: evaluate models on the user's own hardware.
• CLI-first: the terminal is the primary maintained product surface.
• Supported means runnable now.
• Open artifacts: every run should be exportable and inspectable.
• Cross-platform pragmatism: no platform should be treated as second-class.

Current limitations

• Some dependencies are required even for local discovery, including aiohttp.
• Benchmark datasets come from Hugging Face datasets/HF Hub on first use, so the first run of an uncached benchmark still needs network access.
• A fully offline run is possible only after the needed benchmark rows have already been cached locally.
• The standard local install is intentionally small; use requirements-dev.txt only if you need the broader development environment.
• Results are only as comparable as the local runtime settings and hardware conditions you keep consistent.

Legacy code

Older experimental evaluator, web UI, and exporter code lives under legacy/. It is kept for history, but the maintained product surface is the CLI-first pipeline documented above.

Short roadmap

• add more locally runnable public benchmarks where apples-to-apples evaluation is possible
• add dedicated harnesses for code and tool-use benchmarks
• improve benchmark normalization and run metadata for publishable leaderboard workflows
• keep the terminal experience first-class instead of bolting on a web UI

Development

Run the unit tests with:

python -m unittest discover -s tests -q