DeepHedge-RL

Constrained Reinforcement Learning for Dynamic Intraday Options Hedging

Research Areas: Reinforcement Learning • Quantitative Finance • Algorithmic Trading • Risk Management • Explainable AI

---

Overview

DeepHedge-RL is a research framework that formulates intraday options trading and dynamic hedging as a Constrained Markov Decision Process (CMDP).

Instead of relying on analytical delta-hedging rules derived from Black–Scholes assumptions, the framework learns risk-aware execution policies directly from historical market data using constrained deep reinforcement learning.

The system combines realistic market frictions, option Greeks, exchange-style margin constraints, multi-horizon reward shaping, and action masking to learn autonomous hedging strategies for multi-leg options portfolios.

---

Why This Project?

Classical option hedging assumes:

• Continuous trading
• Constant volatility
• Frictionless execution
• Infinitely divisible positions

Real markets violate every one of these assumptions.

Transaction costs, discrete strikes, margin requirements, volatility clustering, and liquidity constraints make analytical hedging increasingly suboptimal.

This project investigates a different question:

Can a reinforcement learning agent learn better hedging policies directly from market experience while respecting realistic exchange constraints?

---

Research Contributions

Unlike conventional DRL trading repositories, DeepHedge-RL contributes several novel engineering components.

Constrained Reinforcement Learning Environment

A custom Gymnasium environment modelling:

• Intraday NIFTY options
• SPAN-style margin simulation
• Transaction costs
• Realistic execution constraints
• Multi-leg portfolios

Action-Masked PPO

Illegal actions are removed before policy sampling — the agent never wastes updates learning impossible trades. This significantly improves:

• Exploration
• Sample efficiency
• Policy stability

Multi-Horizon Reward Engineering

Instead of rewarding immediate PnL, the reward integrates:

• Short-term reaction
• Medium-term trend
• Longer holding performance
• Drawdown penalties

…to encourage stable, risk-aware behavior.

Rich State Representation

69-dimensional observation space including:

• Option Greeks (Delta, Gamma, Vega, Theta, Vanna, Volga)
• India VIX
• Parkinson volatility
• Garman-Klass volatility
• Technical indicators
• Portfolio exposure
• Time features
• Risk statistics
• Position state

Dynamic Margin Simulation

Implements an exchange-inspired SPAN-like model using:

• Stressed implied volatility
• Price scans
• Exposure margins

…to enforce realistic capital constraints.

Explainable Reinforcement Learning

Post-hoc SHAP analysis identifies which Greeks, which volatility measures, and which market regimes drive policy decisions.

---

System Architecture

         Historical Market Data
     (NIFTY + India VIX + Greeks)
                │
                ▼
      Feature Engineering Pipeline
                │
                ▼
      69-Dimensional State Vector
                │
                ▼
  Constraint-aware Gymnasium Environment
                │
  ┌─────────────┴─────────────┐
  │                           │
Action Mask              Reward Function
  │                           │
  ▼                           ▼
      Maskable PPO Agent
                │
                ▼
     Portfolio Execution Engine
                │
                ▼
 Margin + Costs + PnL Simulation
                │
                ▼
         Policy Optimization

---

Repository Structure

DeepHedge-RL/
│
├── envs/
│   ├── intraday_option_env.py
│   ├── intraday_option_env_maskPPO.py
│   └── intraday_option_env_maskPPO_v3.py
│
├── utils/
│   ├── black_scholes.py
│   ├── feature_calculator.py
│   └── margin_engine.py
│
├── training/
│   ├── Mask_PPO_train_v3.py
│   ├── PPO_train.py
│   └── A2C_train.py
│
├── evaluation/
│   ├── walk_forward_validation.py
│   ├── explainability_dashboard.py
│   └── Mask_PPO_explainability_test.py
│
├── dashboards/
│
├── notebooks/
│
├── results/
│
└── docs/

---

Methodology

The framework follows the complete research pipeline:

Historical Data
      │
      ▼
Feature Engineering
      │
      ▼
CMDP Formulation
      │
      ▼
Action Masking
      │
      ▼
Maskable PPO
      │
      ▼
Portfolio Simulation
      │
      ▼
Walk-forward Evaluation
      │
      ▼
Explainability Analysis

---

Experimental Setup

Component	Configuration
RL Algorithm	Maskable PPO
Baselines	PPO, A2C
Observation Space	69 Features
Action Space	Discrete(8)
Training Frequency	1 Minute
Underlying	NIFTY50
Auxiliary Data	India VIX
Validation	Walk Forward
Explainability	SHAP

---

Key Results

Metric	Value
Net PnL	₹561,919
Profit Factor	47.66
Win Rate	93.75%
Max Drawdown	₹66,540

Note

These results were obtained on the 2025 out-of-sample test set under the repository's transaction-cost assumptions. They should be interpreted as backtest results rather than evidence of future live trading performance.

---

Explainability

Rather than treating the trained policy as a black box, DeepHedge-RL performs post-hoc SHAP analysis to understand:

• Policy sensitivity
• Feature importance
• Regime dependence

This enables quantitative inspection of learned hedging behavior across different volatility environments.

---

Current Limitations

Limitation
Flat transaction-cost model
No order-book simulation
No latency modelling
Discrete strike selection
Single underlying
Historical backtesting only

Note

Good researchers openly acknowledge limitations. These are areas of active investigation for future work.

---

Future Work

• Continuous action algorithms (SAC / TD3)
• Transformer-based state encoder
• Multi-asset hedging
• Distributional RL
• CVaR optimization
• Offline RL
• Limit-order execution
• Microstructure-aware state representation

---

Citation

@software{skmishra2026deephedgerl,
  title={DeepHedge-RL: Constrained Reinforcement Learning for Dynamic Intraday Options Hedging},
  author={Saksham Mishra},
  year={2026},
  url={https://github.com/Zylus08/DeepHedge-RL}
}