Skip to content

feat(mcp): selective tool registration via --tools / --exclude-tools #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
codemug merged 2 commits into from
May 24, 2026
Merged

feat(mcp): selective tool registration via --tools / --exclude-tools #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7a1f618
chore: sync uv.lock to v1.2.0
codemug May 24, 2026
7b7a7ce
feat(mcp): selective tool registration via --tools and --exclude-tools
codemug May 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 64 additions & 32 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,35 +5,67 @@ All notable changes to this project will be documented in this file. See [Conven
## [Unreleased]

### Features
- Initial release of aiofmp
- Async-first Python client for Financial Modeling Prep API
- Built-in MCP (Model Context Protocol) server
- 160+ MCP tools across 22+ API categories
- CLI interface for easy MCP server management
- Comprehensive test coverage
- Type hints throughout the codebase
- Support for both STDIO and HTTP transports

### API Categories
- Analyst: Financial estimates, ratings, price targets
- Calendar: Earnings, dividends, IPOs, economic events
- Chart: Historical price data and technical analysis
- Company: Company profiles, key metrics, corporate information
- Commodity: Commodity prices, quotes, historical data
- COT: Commitment of Traders reports
- Crypto: Cryptocurrency prices, quotes, market data
- DCF: Discounted Cash Flow valuations
- Directory: Symbol lists, exchanges, sectors, reference data
- Economics: Economic indicators, treasury rates, macro data
- ETF: ETF holdings, performance, analysis
- Forex: Foreign exchange rates and currency data
- Form 13F: Institutional holdings and filings
- Indexes: Stock market indices and performance
- Insider Trades: Insider trading activity and statistics
- Market Performance: Sector performance, market movers, P/E ratios
- News: Financial news, press releases, market updates
- Quote: Real-time quotes, price changes, market data
- Search: Symbol search, company search, stock screening
- Senate: Congressional trading disclosures
- Statements: Financial statements, ratios, metrics
- Technical Indicators: Moving averages, RSI, technical analysis tools
- **MCP server: selective tool registration.** New `--tools` / `--exclude-tools` CLI flags (with `AIOFMP_MCP_TOOLS` / `AIOFMP_MCP_EXCLUDE_TOOLS` env equivalents) let users restrict which of the 177 MCP tools get registered. The spec grammar mixes category-level and per-tool granularity: `chart` or `chart(*)` for a whole category, `chart(get_intraday_1hour,get_historical_price_full)` for specific tools, comma-separated. When both flags are set, the include set is the universe and exclude prunes from it.
- **MCP server: `--list-tools` flag** prints the full inventory of available categories and tool names, then exits. Does not require an API key.

## [1.2.0] - 2026-05-24

### Features
- **Harvester.** New `aiofmp harvest` / `aiofmp harvest-status` CLI commands: a long-running daemon that proactively warms the local Parquet cache for 17 categories on per-category intervals (chart EOD/intraday, statements, news, analyst estimates/snapshots, insider trades, form 13F, commodities, forex, indexes, economics, DCF, technical indicators).
- **Plan-aware operation.** Reads an FMP plan tier (`basic`/`starter`/`premium`/`ultimate`) from `harvester.yaml`, paces requests via a sliding-window per-minute rate limiter, applies a US-only symbol filter on Starter, and auto-disables categories or endpoints that are entirely paywalled on the chosen plan (e.g. `form13f`, `period=quarter` for `key_metrics`, the `1min` intraday timeframe, the `press_releases` news variant).
- **Per-category paywall short-circuit with re-probe TTL.** After N consecutive HTTP 402s within a cycle, the harvester ends the cycle as `PARTIAL` and skips subsequent cycles for ~24h before re-probing — avoids burning requests against a fully-paywalled endpoint.
- **Bandwidth budget tracker.** SQLite-backed monthly ledger attributes bytes to the category that made the request; soft cap pauses categories until next month; hard cap raises `FMPBudgetError`.
- **SQLite state store.** Per-cycle bookkeeping, last-seen-date checkpoints, symbol-universe catalogs with TTL refresh, and the bandwidth ledger live in `<state_dir>/harvester.sqlite`.
- **Symbol catalog.** Lazily discovers six universes (financial symbols, actively trading, ETFs, commodities, forex pairs, indexes) and refreshes them on a configurable interval; supports payload-aware filters so the indexes universe can drop non-USD listings using FMP's `currency` field.
- **Caching pattern coverage.** New `PAGE_WALK` temporal pattern in `CachedClient` (harvester-write, user-read) for analyst estimates, insider trades, and form 13F. New `SnapshotStore` for the P4 single-row-per-entity pattern used by analyst ratings and DCF.
- **Multi-category cache sharing.** Chart endpoints (`historical-price-eod/*`, intraday) are now registered under `commodity`, `forex`, and `indexes` categories too, so the shared FMP wire endpoint hits the same Parquet store regardless of which SDK category was called.
- **Bandwidth callback + ContextVar.** `FmpClient.on_response_size` reports each response body size, with `current_harvest_category` ContextVar attributing the bytes to the category that initiated the call (or `"user"` outside a harvester cycle).
- **`aiofmp` CLI group.** Existing `aiofmp-mcp-server` entry point is preserved; new `aiofmp` umbrella command wraps `harvest` and `harvest-status`.

### Fixes
- Atomic Parquet writes (write to `.tmp`, then rename) so an interrupted flush can't leave a 0-byte or partially-written file in the cache.
- Recover from 0-byte / corrupt Parquet files on read (delete and refetch) rather than erroring out.
- Sanitize Parquet records before write: stringify columns containing integers outside the float64-safe range (|x| > 2^53), stringify mixed string/numeric columns that appear during append, and null out empty struct values (`{"data": {}}`) that pyarrow can't infer a schema for.
- Per-minute rate-limit margin (~17%) under the documented plan caps to absorb residual server-side 429s.
- 429 and 5xx are transparently retried inside `_make_request` with bounded backoff; everything else (auth, paywall, parse, budget) propagates immediately so callers can apply per-exception policy.
- Per-category retry policy (`retry.on_429`, `retry.on_5xx`) wraps each `run_cycle` independently of the inner client retries.
- Cooperative cancellation: harvester cycles check `should_stop()` between symbols and pages so SIGINT propagates promptly.
- Statements safety-net seeds its checkpoint on first run and is bookkept under a separate `statements_safetynet` category row in the state store.
- News pagination uses a safety cap and short-circuits on empty pages.
- Symbol-catalog refresh treats permanent (4xx other than 429/402) errors as terminal and does not retry them.
- Use FMP's own economic-indicator names (`GDP`, `realGDP`, `CPI`, ...) — FRED codes like `UNRATE` / `FEDFUNDS` return `"Invalid name"`.
- Correct `financial_symbols` endpoint path.
- Quiet log lines for expected paywalls (info-level skip instead of warning per item) and at-startup category auto-disables.
- `aiofmp` CLI entry point is registered.

### Tests
- End-to-end smoke test covering all 17 harvester categories.
- Live-run regression coverage for paywall-memory paths, the mixed-column sanitization, and the 5xx retry policy.

### Docs
- Harvester README section plus `examples/harvester.example.yaml` tuned for Starter plan defaults.

## [1.1.0] - 2026-03-05

### Features
- **CachedClient.** A drop-in caching wrapper around `FmpClient`. Intercepts cacheable time-series methods, stores data in local Parquet files, and only fetches missing date ranges from the API on subsequent calls.
- Supported caching patterns:
- **Date-range endpoints** (chart, economics, calendar, news, technical indicators, ...): gap detection fetches only the missing slices.
- **Period-based endpoints** (income statements, balance sheets, ratios, ...): fetches fresh data and merges with stored history by date.
- **Non-cacheable endpoints** (profiles, quotes, search, ...): pass through directly to the FMP API.
- Pluggable `StorageBackend` interface; ships with `ParquetStorage` (default location: `~/.aiofmp/cache`, override via `AIOFMP_CACHE_FILE_PATH`).
- `EndpointRegistry` describes which SDK methods are cacheable, the temporal pattern, the storage key, and the response date field.
- MCP server gains an `--cached` flag / `AIOFMP_CACHED` env var that switches the underlying client to `CachedClient` so MCP tool calls benefit from the same cache.

## [1.0.0] - 2025-10-21

### Features
- Initial release of aiofmp.
- Async-first Python client for the Financial Modeling Prep API built on asyncio + aiohttp.
- Category-based organization with 22 API categories: analyst, calendar, chart, commodity, company, COT, crypto, DCF, directory, economics, ETF, forex, form 13F, indexes, insider trades, market performance, news, quote, search, senate, statements, technical indicators.
- Built-in MCP (Model Context Protocol) server (`aiofmp-mcp-server`) exposing 177 tools to AI assistants via FastMCP.
- Both STDIO and HTTP transports for the MCP server.
- Comprehensive error handling with typed exceptions: `FMPError`, `FMPAuthenticationError`, `FMPRateLimitError`, `FMPResponseError`, `FMPServerError`, `FMPPaywallError`.
- Connection pooling, configurable timeouts, exponential-backoff retries, and a per-process concurrent-request semaphore.
- Full type hints throughout the codebase.
- 500+ unit tests covering every category and the MCP tool layer.
45 changes: 45 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,39 @@ aiofmp-mcp-server --cached

When `--cached` is enabled, the MCP server uses `CachedClient` under the hood. Time-series API calls are cached locally in Parquet files so repeated queries for the same historical data don't consume API quota. Cache directory defaults to `~/.aiofmp/cache` (override with `AIOFMP_CACHE_FILE_PATH`).

#### Selective Tool Registration

By default the server exposes 177 tools across 22 categories. To reduce the surface area an AI agent has to reason about, restrict which tools are registered using `--tools` (allowlist) and/or `--exclude-tools` (denylist). The spec grammar mixes category-level and per-tool selection:

| Spec | Meaning |
|---|---|
| `chart` | all tools in the chart category |
| `chart(*)` | same — explicit form |
| `chart(get_intraday_1hour)` | just that one chart tool |
| `chart(get_intraday_1hour,get_historical_price_full)` | exactly those chart tools |
| `*` | every tool in every category (default) |

A spec is a comma-separated list of these entries. When both flags are set, `--tools` defines the universe and `--exclude-tools` prunes from it.

```bash
# Only chart, quote, and search are visible to the AI
aiofmp-mcp-server --tools "chart(*),quote(*),search(*)"

# A single tool from quote, all of search
aiofmp-mcp-server --tools "quote(get_stock_quote),search"

# Everything except form13f, senate, and chart.get_intraday_1min
aiofmp-mcp-server --exclude-tools "form13f,senate,chart(get_intraday_1min)"

# Allowlist with a carve-out
aiofmp-mcp-server --tools "chart(*),quote(*)" --exclude-tools "chart(get_intraday_1min)"

# Discover available categories and tool names
aiofmp-mcp-server --list-tools
```

Environment variable equivalents (useful for Claude Desktop configs): `AIOFMP_MCP_TOOLS` and `AIOFMP_MCP_EXCLUDE_TOOLS`. Invalid category or tool names fail fast at startup with the valid list printed.

#### Claude Desktop Integration

Add to your Claude Desktop configuration (`claude_desktop_config.json`):
Expand Down Expand Up @@ -237,6 +270,13 @@ aiofmp-mcp-server --text-content
# Enable local caching
aiofmp-mcp-server --cached

# Restrict the tools that get registered
aiofmp-mcp-server --tools "chart(*),quote(get_stock_quote),search"
aiofmp-mcp-server --exclude-tools "form13f,senate"

# List available categories and tools
aiofmp-mcp-server --list-tools

# All options
aiofmp-mcp-server --transport http --host localhost --port 3000 --log-level INFO --api-key your_key --text-content --cached
```
Expand All @@ -249,6 +289,9 @@ aiofmp-mcp-server --transport http --host localhost --port 3000 --log-level INFO
- `--api-key`: FMP API key (can also be set via `FMP_API_KEY` environment variable)
- `--text-content`: Include text content alongside structured content in MCP tool responses (default: text content is empty when structured content is present)
- `--cached`: Enable CachedClient to cache time-series data locally in Parquet files, minimizing API calls (default: off). Cache dir: `~/.aiofmp/cache` (override with `AIOFMP_CACHE_FILE_PATH`)
- `--tools`: Restrict which MCP tools are registered. Spec syntax: `category` or `category(*)` for a whole category, `category(tool1,tool2)` for specific tools; comma-separated. Env: `AIOFMP_MCP_TOOLS`. Default: all tools.
- `--exclude-tools`: Same syntax as `--tools`, but prunes from the include set. Env: `AIOFMP_MCP_EXCLUDE_TOOLS`.
- `--list-tools`: Print available categories and tool names, then exit. Does not require `FMP_API_KEY`.

### Available API Categories

Expand Down Expand Up @@ -397,6 +440,8 @@ See `docs/superpowers/specs/2026-05-15-harvester-v2-design.md` for the full desi
| `MCP_LOG_LEVEL` | Logging level | `INFO` | No |
| `AIOFMP_CACHED` | Enable CachedClient (`true`/`false`) | `false` | No |
| `AIOFMP_CACHE_FILE_PATH` | Cache directory for Parquet files | `~/.aiofmp/cache` | No |
| `AIOFMP_MCP_TOOLS` | Restrict MCP tools (allowlist; same grammar as `--tools`) | unset | No |
| `AIOFMP_MCP_EXCLUDE_TOOLS` | Prune MCP tools (denylist; same grammar as `--exclude-tools`) | unset | No |

### MCP Server Modes

Expand Down
65 changes: 65 additions & 0 deletions aiofmp/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,34 @@ def cli() -> None:
default=False,
help="Enable CachedClient: cache time-series data locally in Parquet files to minimize API calls. Cache dir defaults to ~/.aiofmp/cache (override with AIOFMP_CACHE_FILE_PATH).",
)
@click.option(
"--tools",
"tools_spec",
default=None,
help=(
"Restrict which MCP tools are registered. Spec syntax: "
"'category' or 'category(*)' for all tools in a category, "
"'category(tool1,tool2)' for specific tools, comma-separated. "
"Example: --tools 'chart(*),quote(get_stock_quote),search'. "
"Use --list-tools to discover names. Env: AIOFMP_MCP_TOOLS."
),
)
@click.option(
"--exclude-tools",
"exclude_tools_spec",
default=None,
help=(
"Same syntax as --tools, but prunes from the include set. "
"Useful with or without --tools. Env: AIOFMP_MCP_EXCLUDE_TOOLS."
),
)
@click.option(
"--list-tools",
"list_tools",
is_flag=True,
default=False,
help="Print available categories and tools, then exit.",
)
def mcp_server(
transport: str,
host: str,
Expand All @@ -66,6 +94,9 @@ def mcp_server(
api_key: str | None,
text_content: bool,
cached: bool,
tools_spec: str | None,
exclude_tools_spec: str | None,
list_tools: bool,
):
"""
Start the aiofmp MCP server.
Expand Down Expand Up @@ -95,6 +126,40 @@ def mcp_server(
# Set logging level
logging.getLogger().setLevel(getattr(logging, log_level.upper()))

# --list-tools is a discovery short-circuit; no server start, no API key needed.
if list_tools:
from .mcp_selection import format_inventory

click.echo(format_inventory())
return

# Validate tool selection specs early so a bad CLI value errors out before
# we open any client sessions. We re-parse them inside register_tools(),
# but doing it here lets click report errors with proper exit codes.
if tools_spec is not None or exclude_tools_spec is not None:
from .mcp_selection import compute_selection, get_tool_inventory, parse_spec

inventory = get_tool_inventory()
try:
include = parse_spec(tools_spec, inventory) if tools_spec else None
exclude = (
parse_spec(exclude_tools_spec, inventory)
if exclude_tools_spec
else None
)
effective = compute_selection(include, exclude, inventory)
except ValueError as e:
raise click.UsageError(str(e)) from e
if not effective:
raise click.UsageError(
"Tool selection is empty after combining --tools and --exclude-tools; "
"no tools would be registered."
)
if tools_spec:
os.environ["AIOFMP_MCP_TOOLS"] = tools_spec
if exclude_tools_spec:
os.environ["AIOFMP_MCP_EXCLUDE_TOOLS"] = exclude_tools_spec

# Set API key if provided
if api_key:
os.environ["FMP_API_KEY"] = api_key
Expand Down
Loading
Loading