RegimeShift

Institutional-Style Quantitative Research Framework for Market Regime Detection & Dynamic Asset Allocation

A production-quality quantitative research framework that detects latent market regimes using Hidden Markov Models and dynamically reallocates a multi-asset portfolio based on the prevailing market state.

---

Overview

Financial markets rarely behave the same way over long periods. Bull markets, crises, sideways markets, and high-volatility periods all exhibit fundamentally different statistical characteristics.

RegimeShift is a complete quantitative research framework that identifies these hidden market regimes using Gaussian Hidden Markov Models (HMMs) and dynamically adjusts portfolio allocations based on the inferred market state.

The framework emphasizes research integrity over backtest performance, incorporating strict temporal validation, transaction cost modeling, statistical model selection, and automated report generation.

---

Key Features

• Hidden Markov Model based market regime detection
• Automatic HMM model selection using AIC/BIC
• 15+ lookahead-free engineered market features
• Ledoit-Wolf covariance estimation for robust portfolio optimization
• Dynamic regime-conditioned portfolio allocation
• Mean-Variance Optimization using CVXPY
• Transaction cost and turnover-aware optimization
• Strict Walk-Forward Validation
• Bootstrap statistical significance testing
• Feature explainability (Permutation Importance, Mutual Information, ANOVA)
• Publication-quality visualization suite
• Automated academic-style research report generation

---

Research Pipeline

                    Yahoo Finance
                          │
                          ▼
                Multi-Asset Data Loader
                          │
                          ▼
                 Data Cleaning & Alignment
                          │
                          ▼
              Feature Engineering Pipeline
          (15+ Lookahead-Free Market Features)
                          │
                          ▼
            Automatic Model Selection (AIC/BIC)
                          │
                          ▼
              Gaussian Hidden Markov Model
                          │
                          ▼
             Regime Characterization Layer
      (Returns, Volatility, Covariance, Persistence)
                          │
                          ▼
            Dynamic Portfolio Optimization
           (CVXPY Mean-Variance Optimization)
                          │
                          ▼
             Walk-Forward Cross Validation
                          │
                          ▼
          Performance Analysis & Diagnostics
                          │
                          ▼
      Publication Figures + Academic Report

---

Sample Outputs

The framework automatically generates publication-quality visualizations.

Market Regime Timeline

---

Portfolio Performance

---

Regime Transition Matrix

---

Benchmark Comparison

---

Example Results

Latest Walk-Forward Evaluation

Metric	RegimeShift
Walk-Forward Folds	37
Selected Hidden States	5
CAGR	8.90%
Sharpe Ratio	0.91
Maximum Drawdown	-15.93%
Bootstrap Outperformance Probability	68.6%
Bootstrap p-value	0.314

Interpretation

Although RegimeShift achieved a higher Sharpe Ratio than the Buy & Hold benchmark, bootstrap analysis indicates that the observed improvement is not statistically significant at the 5% level, demonstrating the framework's commitment to honest and reproducible quantitative research rather than performance marketing.

---

Asset Universe

Asset	Ticker	Role
NIFTY 50	^NSEI	Equity Exposure
Gold ETF	GLD	Inflation Hedge
Long Treasury ETF	TLT	Defensive Asset

The asset universe is fully configurable through config/settings.yaml.

---

Project Structure

RegimeShift/

├── config/
│   ├── settings.yaml
│   └── logging.yaml
│
├── data/
│   ├── loader.py
│   ├── preprocessor.py
│   └── cache/
│
├── features/
│   ├── market_features.py
│   └── feature_pipeline.py
│
├── models/
│   ├── hmm_model.py
│   └── model_selection.py
│
├── regimes/
│   ├── classifier.py
│   ├── regime_statistics.py
│   └── transition_analysis.py
│
├── portfolio/
│   ├── allocator.py
│   ├── optimizer.py
│   └── constraints.py
│
├── execution/
│   └── rebalance.py
│
├── validation/
│   └── walk_forward.py
│
├── metrics/
│   └── performance.py
│
├── visualization/
│   └── plots.py
│
├── reports/
│   ├── generator.py
│   ├── figures/
│   └── results/
│
├── utils/
│   ├── logger.py
│   └── seed.py
│
├── main.py
├── requirements.txt
└── README.md

---

Installation

Clone the repository

git clone <repository-url>

cd RegimeShift

Create a virtual environment

python -m venv .venv

Activate

Windows

.venv\Scripts\activate

Linux / macOS

source .venv/bin/activate

Install dependencies

pip install -r requirements.txt

---

Quick Start

Run the complete research pipeline

python main.py

Configuration validation

python main.py --dry-run

Download data only

python main.py --phase data

Run model selection only

python main.py --phase model_select

Force fresh Yahoo Finance download

python main.py --force-download

Skip plotting

python main.py --no-plots

Skip report generation

python main.py --no-report

---

Configuration

The framework is fully configurable through

config/settings.yaml

Example

assets:
  tickers:
    - "^NSEI"
    - "GLD"
    - "TLT"

hmm:
  n_states_grid:
    - 2
    - 3
    - 4
    - 5
  selection_criterion: bic
  n_restarts: 10

portfolio:
  risk_aversion: 1.0
  turnover_penalty: 0.5

execution:
  transaction_cost_bps: 7

walk_forward:
  train_years: 2
  test_months: 6
  step_months: 3

---

Research Integrity

The framework is explicitly designed to minimize common sources of bias in quantitative finance.

Potential Issue	Mitigation
Lookahead Bias	All rolling features shifted by one period
Data Leakage	Train-only scaler fitting and strict temporal validation
Overfitting	Model selected using BIC instead of backtest performance
Covariance Instability	Ledoit-Wolf shrinkage estimator
Unrealistic Execution	Transaction costs and turnover penalties included
Regime Labelling Bias	Data-driven statistical labeling

---

Performance Metrics

The framework automatically computes:

Return Metrics

• Total Return
• CAGR
• Annual Return

Risk Metrics

• Annual Volatility
• Maximum Drawdown
• Average Drawdown
• Ulcer Index

Risk-Adjusted Metrics

• Sharpe Ratio
• Sortino Ratio
• Calmar Ratio
• Recovery Factor

Tail Risk

• CVaR (95%)
• Expected Shortfall
• Skewness
• Kurtosis
• Tail Ratio

Execution Metrics

• Portfolio Turnover
• Rebalance Frequency
• Average Holding Period

Rolling Metrics

• Rolling Sharpe
• Rolling Volatility
• Rolling Drawdown

---

Generated Outputs

After execution the framework produces

reports/

├── research_report.md
├── regimeshift.log
│
├── figures/
│   ├── Regime Timeline
│   ├── Posterior Probabilities
│   ├── Transition Matrix
│   ├── Stationary Distribution
│   ├── Portfolio Allocation
│   ├── Equity Curves
│   ├── Rolling Drawdowns
│   ├── Rolling Sharpe
│   ├── Benchmark Comparison
│   ├── Turnover Timeline
│   ├── Regime Duration Distribution
│   ├── Model Selection
│   └── Correlation Matrix
│
└── results/
    ├── Performance Tables
    ├── Transition Matrices
    ├── Weight History
    ├── Portfolio Returns
    └── Model Selection Results

---

Technical Stack

Library	Purpose
hmmlearn	Hidden Markov Models
CVXPY	Portfolio Optimization
scikit-learn	Feature Scaling & Ledoit-Wolf Covariance
pandas	Data Manipulation
NumPy	Numerical Computing
SciPy	Statistical Analysis
yfinance	Market Data
matplotlib	Visualization
seaborn	Publication Graphics
joblib	Model Serialization
PyYAML	Configuration

---

Design Principles

RegimeShift was developed around five core principles:

• Statistical rigor over headline performance
• Strict elimination of lookahead bias
• Fully reproducible research workflows
• Modular and extensible architecture
• Honest reporting of both strengths and limitations

---

Current Limitations

Current implementation focuses on:

• Daily frequency data
• Gaussian Hidden Markov Models
• Long-only portfolio optimization
• Historical backtesting
• Static feature engineering

These design choices prioritize research reproducibility while leaving room for future extensions.

---

Future Work

Potential research directions include:

• Hidden Semi-Markov Models (HSMM)
• Bayesian Hidden Markov Models
• Bayesian Online Change Point Detection
• Ensemble Regime Detection
• Online Learning
• Reinforcement Learning Allocation
• Macroeconomic Feature Integration
• Alternative Risk Measures
• Intraday Regime Detection

---

References

1. Hamilton, J. D. (1989). A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle.

2. Rabiner, L. R. (1989). A Tutorial on Hidden Markov Models.

3. Markowitz, H. (1952). Portfolio Selection.

4. Ledoit, O., & Wolf, M. (2004). A Well-Conditioned Estimator for Large-Dimensional Covariance Matrices.

5. Ang, A., & Bekaert, G. (2002). International Asset Allocation with Regime Shifts.

6. Guidolin, M., & Timmermann, A. (2007). Asset Allocation under Multivariate Regime Switching.

---

Disclaimer

This project is intended for research and educational purposes only.

Backtests are historical simulations and should not be interpreted as evidence of future investment performance.

Nothing contained in this repository constitutes financial or investment advice.