time series forecasting with timesfm

.gitignore

@@ -1,3 +1,5 @@
+.DS_Store
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[codz]

2026-04-time-series-forecasting-timesfm/README.md

@@ -0,0 +1,115 @@
+# QuestDB → TimesFM: Time-Series Data Loading & Forecasting
+
+This tutorial demonstrates how to efficiently load time-series data from QuestDB into Python, then run forecasts using Google's TimesFM foundation model.
+
+**Focus**: Data pipeline patterns. The forecasting is just to show the complete workflow.
+
+## What's Inside
+
+| File | Purpose |
+|------|---------|
+| `steps/method1_direct_query.py` | Load data via REST API and ConnectorX |
+| `steps/method2_parquet_export.py` | Export data to Parquet files |
+| `steps/method3_parquet_partitions.py` | Read QuestDB partitions directly |
+| `steps/forecast.py` | Run TimesFM forecasts on volume & volatility |
+
+## The Use Case
+
+We forecast two operationally useful metrics from crypto trade data:
+
+1. **Volume** – When will liquidity be available? (execution timing)
+2. **Volatility** – How much is price moving? (risk management)
+
+We intentionally avoid price prediction—forecasting "where will BTC be in an hour" isn't credible. Volume and volatility forecasting, however, serve real needs.
+
+## Quick Start
+
+> **Requires Python 3.10+**
+
+```bash
+# 1. Clone and setup
+git clone https://github.com/questdb/blog-examples.git
+cd blog-examples/2025-pytorch-questdb-timesfm
+
+# 2. Create environment
+python3.11 -m venv .venv
+source .venv/bin/activate
+
+# 3. Install base dependencies
+pip install pandas pyarrow requests torch
+
+# 4. Install TimesFM from source (pip package is outdated)
+git clone https://github.com/google-research/timesfm.git
+cd timesfm
+pip install -e ".[torch]"
+cd ..
+
+# 5. Run the pipeline
+python steps/method1_direct_query.py   # Test QuestDB connection
+python steps/method2_parquet_export.py # Export data to Parquet
+python steps/forecast.py               # Run forecasts
+```
+
+## Data Source
+
+Uses the `trades` table from QuestDB's public demo: `https://demo.questdb.io`
+
+Raw trades are aggregated into 1-minute OHLCV bars:
+
+```sql
+SELECT 
+    timestamp,
+    first(price) as open,
+    max(price) as high,
+    min(price) as low,
+    last(price) as close,
+    sum(amount) as volume,
+    count() as trade_count
+FROM trades
+WHERE symbol = 'BTC-USDT'
+SAMPLE BY 1m
+ALIGN TO CALENDAR
+```
+
+## Three Data Loading Methods
+
+| Method | Best For | How |
+|--------|----------|-----|
+| **REST API** | Exploration, small data | `requests.get("/exp?fmt=csv")` |
+| **ConnectorX** | Large results (local QuestDB) | PostgreSQL wire protocol |
+| **Parquet Export** | Reproducible pipelines | `requests.get("/exp?fmt=parquet")` |
+| **Direct Partitions** | Terabyte-scale | Read QuestDB's Parquet files directly |
+
+> **Note**: ConnectorX requires PostgreSQL port access (8812). The public demo only exposes REST API, so ConnectorX works with local/self-hosted QuestDB only.
+
+## Project Structure
+
+```
+├── config/settings.py       # Configuration (URLs, symbols, params)
+├── src/
+│   ├── data_loader.py       # Three data loading methods
+│   └── forecaster.py        # TimesFM wrapper
+├── steps/
+│   ├── method1_direct_query.py
+│   ├── method2_parquet_export.py
+│   ├── method3_parquet_partitions.py
+│   └── forecast.py
+├── dataset/                 # Exported Parquet files
+├── outputs/                 # Forecast results (CSV)
+└── BLOG_POST.md             # Full tutorial writeup
+```
+
+## Notes
+
+- **Python 3.10+ required** – TimesFM doesn't support older versions
+- **TimesFM must be installed from source** – The PyPI package is outdated
+- **Model downloads ~900MB on first run** – Cached in `~/.cache/huggingface/`
+- **GPU optional** – CPU inference works fine, just slower
+
+## Honest Caveats
+
+1. **Foundation models aren't magic.** TimesFM gives a quick baseline, but tuned domain-specific models may perform better.
+
+2. **Volume is hard to forecast.** It's driven by news, large orders, and sentiment—all unpredictable. Wide confidence intervals are expected.
+
+3. **This is a data pipeline tutorial.** Whether these forecasts are useful for your application requires proper backtesting.

2026-04-time-series-forecasting-timesfm/config/settings.py

@@ -0,0 +1,26 @@
+from pathlib import Path
+
+# QuestDB connection settings
+# REST API - points to public demo instance (works out of the box)
+QUESTDB_HTTP = "https://demo.questdb.io"
+
+# PostgreSQL wire protocol - points to local QuestDB (for ConnectorX)
+# The demo instance doesn't expose port 8812 externally, so this requires local QuestDB
+QUESTDB_PG = "postgresql://admin:quest@localhost:8812/qdb"
+
+# For local QuestDB with Parquet partitions (Method 3)
+QUESTDB_DATA_DIR = Path("/var/lib/questdb/db")
+
+# Paths
+PROJECT_ROOT = Path(__file__).parent.parent
+DATASET_DIR = PROJECT_ROOT / "dataset"
+OUTPUT_DIR = PROJECT_ROOT / "outputs"
+
+# Data settings
+TABLE_NAME = "trades"
+SYMBOL = "BTC-USDT"
+
+# TimesFM settings
+TIMESFM_MODEL = "google/timesfm-2.5-200m-pytorch"
+MAX_CONTEXT = 512      # How much history to use
+FORECAST_HORIZON = 60  # How far ahead to forecast (in minutes)

2026-04-time-series-forecasting-timesfm/pyproject.toml

@@ -0,0 +1,26 @@
+[project]
+name = "pytorch-questdb-timesfm"
+version = "0.1.0"
+description = "Tutorial: QuestDB time-series data with Google's TimesFM foundation model"
+requires-python = ">=3.10"
+dependencies = [
+    "pandas>=2.0.0",
+    "pyarrow>=15.0.0",
+    "requests>=2.31.0",
+    "torch>=2.0.0",
+]
+
+[project.optional-dependencies]
+# NOTE: TimesFM must be installed from source, not pip
+# git clone https://github.com/google-research/timesfm.git
+# cd timesfm && pip install -e ".[torch]"
+fast = [
+    "connectorx>=0.3.0",  # Faster queries via PostgreSQL protocol
+]
+
+[tool.poe.tasks]
+method1  = { cmd = "python steps/method1_direct_query.py", help = "Demo: Direct SQL queries" }
+method2  = { cmd = "python steps/method2_parquet_export.py", help = "Demo: Parquet export via API" }
+method3  = { cmd = "python steps/method3_parquet_partitions.py", help = "Demo: Direct partition access" }
+forecast = { cmd = "python steps/forecast.py", help = "Run TimesFM forecasting" }
+all      = ["method1", "method2", "forecast"]

2026-04-time-series-forecasting-timesfm/requirements.txt

@@ -0,0 +1,10 @@
+# Core dependencies
+pandas>=2.0.0
+pyarrow>=15.0.0
+requests>=2.31.0
+torch>=2.0.0
+connectorx>=0.3.0
+
+# TimesFM must be installed separately from source:
+#   git clone https://github.com/google-research/timesfm.git
+#   cd timesfm && pip install -e ".[torch]"

2026-04-time-series-forecasting-timesfm/src/data_loader.py

@@ -0,0 +1,165 @@
+"""
+Three methods to load data from QuestDB into Python.
+"""
+import pandas as pd
+import requests
+from io import StringIO
+from pathlib import Path
+
+from config.settings import QUESTDB_HTTP, QUESTDB_PG
+
+
+# =============================================================================
+# Method 1: Direct Queries
+# =============================================================================
+
+def query_rest_api(query: str) -> pd.DataFrame:
+    """
+    Query QuestDB via REST API, return DataFrame.
+
+    Simple and works everywhere. Best for exploration and smaller datasets.
+    """
+    response = requests.get(
+        f"{QUESTDB_HTTP}/exp",
+        params={"query": query, "fmt": "csv"}
+    )
+    response.raise_for_status()
+    return pd.read_csv(StringIO(response.text))
+
+
+def query_connectorx(query: str) -> pd.DataFrame:
+    """
+    Query QuestDB via PostgreSQL wire protocol using ConnectorX.
+
+    Faster than REST for larger results. Requires: pip install connectorx
+
+    Note: Uses redshift:// driver because QuestDB's PG wire protocol is 
+    compatible with Redshift's simpler dialect.
+    """
+    import connectorx as cx
+    # Replace postgresql:// with redshift:// for QuestDB compatibility
+    conn = QUESTDB_PG.replace("postgresql://", "redshift://")
+    return cx.read_sql(conn, query)
+
+
+# =============================================================================
+# Method 2: Parquet Export via REST API
+# =============================================================================
+
+def export_to_parquet(query: str, output_path: Path) -> Path:
+    """
+    Export query results to Parquet file via QuestDB REST API.
+
+    Falls back to CSV if Parquet export is disabled on the server.
+    Best for reproducible datasets.
+    """
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+
+    # Try native Parquet export
+    response = requests.get(
+        f"{QUESTDB_HTTP}/exp",
+        params={"query": query, "fmt": "parquet"},
+        stream=True
+    )
+
+    content_type = response.headers.get("content-type", "")
+
+    if response.ok and "parquet" in content_type:
+        with open(output_path, "wb") as f:
+            for chunk in response.iter_content(chunk_size=8192):
+                f.write(chunk)
+        print(f"✓ Exported native Parquet: {output_path}")
+    else:
+        # Fallback: CSV export + pandas conversion
+        print("  Parquet export not available, using CSV fallback...")
+        csv_response = requests.get(
+            f"{QUESTDB_HTTP}/exp",
+            params={"query": query, "fmt": "csv"}
+        )
+        csv_response.raise_for_status()
+        df = pd.read_csv(StringIO(csv_response.text))
+        df.to_parquet(output_path, index=False)
+        print(f"✓ Exported via CSV→Parquet: {output_path}")
+
+    return output_path
+
+
+# =============================================================================
+# Method 3: Direct Parquet Partition Access
+# =============================================================================
+
+def load_parquet_partitions(table_name: str, data_dir: Path):
+    """
+    Load QuestDB Parquet partitions directly via PyArrow.
+
+    Requires:
+    - Local/mounted access to QuestDB data directory
+    - Parquet partition format enabled (o3PartitionFormat = 'PARQUET')
+
+    Best for terabyte-scale data and streaming processing.
+    """
+    import pyarrow.dataset as ds
+
+    table_dir = data_dir / table_name
+    parquet_files = list(table_dir.glob("**/*.parquet"))
+
+    if not parquet_files:
+        raise FileNotFoundError(
+            f"No Parquet files found in {table_dir}. "
+            "Ensure Parquet partition format is enabled in QuestDB."
+        )
+
+    print(f"✓ Found {len(parquet_files)} partition files")
+    return ds.dataset(table_dir, format="parquet")
+
+
+def stream_from_partitions(dataset, columns: list[str], batch_size: int = 100_000):
+    """
+    Stream data from Parquet partitions in batches.
+
+    Memory-efficient for large datasets.
+    """
+    scanner = dataset.scanner(columns=columns, batch_size=batch_size)
+    for batch in scanner.to_batches():
+        yield batch.to_pandas()
+
+
+# =============================================================================
+# Query builders
+# =============================================================================
+
+def get_ohlcv_query(symbol: str = "BTC-USDT", days: int = 7, interval: str = "1m") -> str:
+    """
+    Build query for OHLCV data aggregated at specified interval.
+
+    Returns: timestamp, open, high, low, close, volume
+    """
+    return f"""
+    SELECT 
+        timestamp,
+        first(price) as open,
+        max(price) as high,
+        min(price) as low,
+        last(price) as close,
+        sum(amount) as volume,
+        count() as trade_count
+    FROM trades
+    WHERE symbol = '{symbol}'
+        AND timestamp > dateadd('d', -{days}, now())
+    SAMPLE BY {interval}
+    ALIGN TO CALENDAR
+    """
+
+
+def get_volume_series_query(symbol: str = "BTC-USDT", days: int = 7) -> str:
+    """Build query for volume time series (1-minute intervals)."""
+    return f"""
+    SELECT 
+        timestamp,
+        sum(amount) as volume
+    FROM trades
+    WHERE symbol = '{symbol}'
+        AND timestamp > dateadd('d', -{days}, now())
+    SAMPLE BY 1m
+    ALIGN TO CALENDAR
+    """