Skip to content

f5devcentral/llm-inference-lb

BranchesTags

Repository files navigation

[中文Readme] [模块关系-中文] [程序有效性评估-中文] [算法对比与选择指导]

[Modules relationships] [Program effectiveness evaluation-EN] [Scheduler Algorithm Comparison Analysis and guide]

[LLM Performance improvement test]

F5 LLM Inference Gateway Scheduler

An intelligent scheduler for LLM inference gateway, designed to work with F5 LTM for optimal load balancing based on real-time performance metrics from inference engines.

abstraction-arch

Features

  • Intelligent Scheduling Algorithm: S1,S2. Based on different LLM server metrics
  • Multi-Engine Support: Supports vLLM and SGLang inference engines, including variants (e.g., vllm_ascend, vllm_musa, vllm-mlu)
  • Heterogeneous Engine Pool: engine_type: auto enables vLLM + SGLang mixed pools with per-member automatic engine detection
  • Real-time Monitoring: Automatically fetches F5 Pool members and inference engine performance metrics
  • High Availability Design: Asynchronous architecture with concurrent processing support
  • RESTful API: Provides standard HTTP interfaces
  • Configuration Hot Reload: Supports runtime configuration updates
  • Comprehensive Logging: Detailed debugging and runtime logs
  • Weighted Random Selection: Score-based probabilistic selection algorithm
  • Manual or automatic fallback mechanisms: can be applied in scenarios such as load balancing algorithm fallback for inference servers, service backup, maintenance control, version A/B rollout, and cross-region inference traffic scheduling.
  • Performance Analysis: Provides selection process simulation and probability analysis interfaces

Project Structure

scheduler-project/
├── main.py                 # Main program entry point
├── config/
│   ├── __init__.py
│   ├── config_loader.py    # Configuration file loader module
│   └── scheduler-config.yaml  # Configuration file
├── core/
│   ├── __init__.py
│   ├── models.py           # Data model definitions
│   ├── f5_client.py        # F5 API client
│   ├── metrics_collector.py # Metrics collection module
│   ├── score_calculator.py  # Score calculation module
│   └── scheduler.py        # Scheduler core logic
├── api/
│   ├── __init__.py
│   └── server.py           # API server
├── utils/
│   ├── __init__.py
│   ├── logger.py           # Logging utilities
│   └── exceptions.py       # Custom exceptions
├── tests/                  # Test files
├── requirements.txt        # Project dependencies
└── README.md              # Project documentation

Modules Relationships

Check here for detailed architecture.

Installation and Deployment

Prerequisites

Set up F5 BIG-IP:

  • http standard vs
  • optional: session persistence (source ip or based on session/cookie, depends your real case)
  • apply the irule in the docs/F5-irule (change related variables to yours)
  • set default inference pool and/or fallback inference pool(depends your needs)
  • set least conn LB for the pool
  • create a guest account that can list these pools on the BIG-IP

Set up your inference engine correctly. Currently offcial support vLLM, SGlang. For xInference it is WIP for now.

1. Environment Requirements

  • Python 3.10+
  • F5 LTM device access permissions
  • Inference engine services (vLLM or SGLang)

2. Install Dependencies

pip install -r requirements.txt

3. Configuration File

Configuration file:

config/scheduler-config.yaml

Edit the configuration file to set F5 connection information and Pool configuration:

global:
  interval: 5                    # Configuration hot reload check interval (seconds)
  api_port: 8080                # API service port
  api_host: 0.0.0.0             # API service listening address
  log_level: INFO               # Log level

f5:
  host: 192.168.1.100           # F5 device IP (required)
  port: 443                     # F5 management port
  username: admin               # F5 username
  password_env: F5_PASSWORD     # F5 password environment variable

scheduler:
  pool_fetch_interval: 10       # Pool member fetch interval (seconds)
  metrics_fetch_interval: 3000  # Metrics collection interval (milliseconds)

modes:
  - name: s1_enhanced           # Algorithm mode name
    w_a: 0.1                    # Waiting queue weight
    w_b: 0.9                    # Cache usage weight

pools:
  - name: llm-pool-1            # Pool name (required)
    partition: Common           # Partition name
    engine_type: vllm           # Engine type (required)
    fallback:                   # Fallback configuration (optional)
      pool_fallback: false      # Pool-level fallback switch
      member_running_req_threshold: 20.0    # Running requests threshold
      member_waiting_queue_threshold: 15.0  # Waiting queue threshold
    metrics:
      schema: http              # Protocol type
      #port: 5001								# when metrics port is different to the port of F5 pool members
      path: /metrics            # Metrics path
      timeout: 4                # Request timeout
# Engine variants metrics keys configuration (optional)
# Use this to support vLLM/SGLang variants with different metrics key names
engines_metrics_keys:
  vllm_ascend:                 # Huawei Ascend variant
    waiting_queue: vllm:num_requests_waiting
    cache_usage: vllm:kv_cache_usage_perc  # Ascend uses kv_cache instead of gpu_cache
    running_req: vllm:num_requests_running
  vllm_musa:                   # Moore Threads variant
    cache_usage: vllm:gpu_cache_usage_perc
  vllm-mlu:                    # Cambricon variant  
    cache_usage: vllm:mlu_cache_usage_perc

Known Issue 1:

When configuring the specified port under metrics, it means that the scheduler no longer uses the port in the F5 Pool members. Since the configuration file only allows the definition of one port at this time, if the IP in the F5 Pool members is also the same IP, This will cause the scheduler to obtain metrics for different pool members with the same IP and the same port, which will cause problems. Therefore, if the metrics port needs to be specified, the IP in the pool member of F5 must be different.

4. Set Environment Variables

export F5_PASSWORD="your_f5_password"
export METRIC_PWD="your_metrics_password"  # If needed

# Optional: Log file path configuration (for non-Docker deployment)
export LOG_FILE_PATH="/var/log/f5-scheduler/scheduler.log"  # Custom log file path

Log File Path Configuration

Optional Environment Variable: LOG_FILE_PATH

  • If set: The scheduler will write logs to the specified file path

    export LOG_FILE_PATH="/var/log/f5-scheduler/scheduler.log"
# Logs will be written to: /var/log/f5-scheduler/scheduler.log

  • If not set: The scheduler will use the default log file path

    # Default log file: scheduler.log (in the current working directory)
# For example: if you run the scheduler from /opt/f5-scheduler/, 
# the log file will be created at /opt/f5-scheduler/scheduler.log

Note: This environment variable is primarily used for non-Docker deployments. For Docker deployments, use the LOG_TO_STDOUT and LOG_FILE_PATH environment variables as described in the Docker Deployment section.

5. Start the Scheduler

python main.py

Docker Deployment

Production Deployment Example (Recommended)

# Build production image
docker build -f Dockerfile.production -t f5-scheduler:latest .

# Run with production configuration (stdout logging - recommended)
docker run -d \
  --name f5-scheduler \
  -p 8080:8080 \
  -v $(pwd)/config/scheduler-config.yaml:/app/config/scheduler-config.yaml:ro \
  -e F5_PASSWORD=your-password \
  -e METRIC_PWD=your-metric-password \ #optional
  -e LOG_TO_STDOUT=true \
  --log-driver json-file \
  --log-opt max-size=100m \
  --log-opt max-file=3 \
  --restart unless-stopped \
  f5-scheduler:latest

Alternative: File Logging

# Run with file logging (if required by your environment)
docker run -d \
  --name f5-scheduler-container \
  -p 8080:8080 \
  -v $(pwd)/config/scheduler-config.yaml:/app/config/scheduler-config.yaml:ro \
  -v $(pwd)/logs:/app/logs \
  -e F5_PASSWORD="your_f5_password" \
  -e METRIC_PWD="your_metrics_password" \
  -e LOG_TO_STDOUT="false" \
  -e LOG_FILE_PATH="/app/logs/scheduler.log" \
  --restart unless-stopped \
  f5-scheduler:latest

Environment Variables

# Required
-e F5_PASSWORD="your_f5_password"                    # F5 device password

# Optional
-e METRIC_PWD="your_metrics_password"                # Metrics service password (optional)
-e LOG_TO_STDOUT="true"                              # Log output method (optional, production only, default: true recommended)
-e LOG_FILE_PATH="/app/logs/scheduler.log"           # Log file path (optional, only used when LOG_TO_STDOUT=false)

Logging Best Practices

Recommended: Use LOG_TO_STDOUT="true" (default) for container deployments because:

  • Follows 12-Factor App principles and container best practices
  • Better integration with Docker/Kubernetes logging systems
  • Easier log collection with centralized logging solutions (ELK, Fluentd, etc.)
  • Use docker logs -f f5-scheduler-container to view logs
  • Better performance (no file I/O overhead)

File logging should only be used when required by specific enterprise environments or legacy log collection systems.

API Interfaces

1. Select Optimal Member

POST /scheduler/select

Function: Select the optimal member based on real-time performance metrics of Pool members

Request Body:

{
  "pool_name": "llm-pool-1",
  "partition": "Common", 
  "members": ["10.10.10.10:8001", "10.10.10.10:8002"]
}

Response:

Successfully selected optimal member:

10.10.10.10:8001

Pool has fallback mode enabled:

fallback

Unable to select optimal member (Pool doesn't exist, empty member list, all members have Score of 0, etc.):

none

Status Codes:

  • 200: Success (includes successful selection, fallback, and unable to select scenarios)
  • 400: Bad request parameters
  • 500: Internal server error

Response Types:

  • Normal Selection: Returns specific member address (e.g., 10.10.10.10:8001)
  • Fallback Mode: Returns string fallback when Pool is configured with pool_fallback: true
  • Unable to Select: Returns string none

Common scenarios when unable to select:

  • Pool does not exist in the scheduler
  • No intersection between requested member list and actual Pool members
  • No members in the Pool
  • All candidate members are filtered out by thresholds

2. Get Single Pool Status

GET /pools/{pool_name}/{partition}/status

Function: Get detailed status information for a specific Pool

Parameters:

  • pool_name: Pool name
  • partition: Partition name

Response:

{
  "name": "llm-pool-1",
  "partition": "Common",
  "engine_type": "vllm",
  "member_count": 2,
  "members": [
    {
      "ip": "10.10.10.10",
      "port": 8001,
      "score": 0.75,
      "metrics": {
        "waiting_queue": 2.0,
        "cache_usage": 0.3
      },
      "detected_variant": "vllm_ascend",
      "detected_engine_type": "vllm",
      "detection_status": "ok"
    },
    {
      "ip": "10.10.10.10",
      "port": 8002,
      "score": 0.82,
      "metrics": {
        "waiting_queue": 1.5,
        "cache_usage": 0.25
      },
      "detected_variant": "vllm",
      "detected_engine_type": "vllm",
      "detection_status": "ok"
    }
  ]
}

GET /pools/{pool_name}/{partition}/status?simple

Function: Get simpele score information for a specific Pool

Parameters:

  • pool_name: Pool name
  • partition: Partition naem
  • simple: query parameter

Response:

127.0.0.1:8001 0.5404
127.0.0.1:8002 0.0000
127.0.0.1:8003 0.2846

3. Get All Pools Status

GET /pools/status

Function: Get status information for all Pools

Response:

{
  "pools": [
    {
      "name": "llm-pool-1",
      "partition": "Common",
      "engine_type": "vllm",
      "member_count": 2,
      "members": [...]
    },
    {
      "name": "llm-pool-2",
      "partition": "Common",
      "engine_type": "sglang",
      "member_count": 3,
      "members": [...]
    }
  ]
}

4. Health Check

GET /health

Function: Check scheduler service health status

Response:

{
  "status": "healthy",
  "message": "Scheduler is running normally"
}

5. Simulate Selection Process

POST /pools/{pool_name}/{partition}/simulate

Function: Simulate multiple selection processes for testing and analysis (test interface)

Parameters:

  • pool_name: Pool name
  • partition: Partition name
  • iterations: Number of simulations (query parameter, default 100)

Request Body:

{
  "pool_name": "llm-pool-1",
  "partition": "Common",
  "members": ["10.10.10.10:8001", "10.10.10.10:8002"]
}

Response:

{
  "results": {
    "10.10.10.10:8001": 45,
    "10.10.10.10:8002": 55
  },
  "iterations": 100
}

6. Advanced Probability Analysis

POST /pools/{pool_name}/{partition}/analyze

Function: Detailed analysis of selection accuracy and probability bias (test interface)

Parameters:

  • pool_name: Pool name
  • partition: Partition name
  • iterations: Number of analyses (query parameter, default 1000)

Request Body:

{
  "pool_name": "llm-pool-1",
  "partition": "Common",
  "members": ["10.10.10.10:8001", "10.10.10.10:8002"]
}

Response:

{
  "member_analysis": {
    "10.10.10.10:8001": {
      "theoretical_probability": 0.4286,
      "actual_probability": 0.4310,
      "selection_count": 431,
      "deviation": 0.0024,
      "deviation_percentage": 0.56
    },
    "10.10.10.10:8002": {
      "theoretical_probability": 0.5714,
      "actual_probability": 0.5690,
      "selection_count": 569,
      "deviation": -0.0024,
      "deviation_percentage": -0.42
    }
  },
  "overall_stats": {
    "total_iterations": 1000,
    "avg_deviation": 0.0024,
    "max_deviation": 0.0024,
    "quality_score": 99.44
  }
}

Complete Configuration Documentation

Global Configuration (global)

Config Item Type Required Default Description
interval Integer No 60 Configuration file hot reload check interval (seconds)
api_port Integer No 8080 API service listening port
api_host String No "0.0.0.0" API service listening address (0.0.0.0 means all interfaces)
log_level String No "INFO" Log level (DEBUG/INFO/WARNING/ERROR/CRITICAL)
log_debug Boolean No false Backward compatible debug switch (used when log_level is not configured)

F5 Configuration (f5)

Config Item Type Required Default Description
host String Yes None F5 device IP address or hostname
port Integer No 443 F5 iControl REST API port
username String No "admin" F5 device login username. The guest role or high.
password_env String No None Environment variable name for F5 password

Scheduler Configuration (scheduler)

Config Item Type Required Default Description
pool_fetch_interval Integer No 10 Interval to fetch Pool members from F5 (seconds)
metrics_fetch_interval Integer No 1000 Interval to collect Metrics from inference engines (milliseconds)

Algorithm Mode Configuration (modes)

Config Item Type Required Default Description
name String No "s1" Algorithm mode name (supports s1 and s2)
w_a Float No 0.5 Waiting queue weight (between 0-1)
w_b Float No 0.5 Cache usage weight (between 0-1)
w_g Float No 0.0 Running requests weight (used in S2 algorithm)

Pool Configuration (pools)

Config Item Type Required Default Description
name String Yes None Pool name, must match Pool name on F5
partition String No "Common" Partition name on F5
engine_type String Yes None Inference engine type: vllm, sglang, or auto (heterogeneous pool)

Fallback Configuration (pools[].fallback)

Config Item Type Required Default Description
pool_fallback Boolean No false Pool-level fallback switch, returns "fallback" when enabled
member_running_req_threshold Float No null Running requests threshold, members are excluded when exceeded
member_waiting_queue_threshold Float No null Waiting queue length threshold, members are excluded when exceeded

Fallback Feature Description:

  • Pool-level fallback: When pool_fallback: true, /scheduler/select API directly returns string "fallback" without any member selection or score calculation
  • Member threshold filtering: Compares against raw metrics values, members exceeding thresholds are excluded from selection
  • Priority: Pool-level fallback has the highest priority; when enabled, member threshold filtering is ignored
  • Threshold comparison: Uses raw collected metric values (not normalized scores) for direct numerical comparison with configured thresholds

Metrics Configuration (pools[].metrics)

Config Item Type Required Default Description
schema String No "http" Protocol type (http/https)
port Integer No null Metrics service port, null means use Pool member's own port
path String No "/metrics" URL path for Metrics service
timeout Integer No 3 HTTP request timeout (seconds)
APIkey String No null API key for Metrics service
metric_user String No null Username for Metrics service
metric_pwd_env String No null Environment variable name for Metrics service password

Engine Variants Configuration (engines_metrics_keys)

Config Item Type Required Default Description
{variant_name} Object No null Variant configuration block (e.g., vllm_ascend, vllm_musa, sglang_xxx)
{variant_name}.waiting_queue String No null Custom waiting queue metric key
{variant_name}.cache_usage String No null Custom cache usage metric key
{variant_name}.running_req String No null Custom running requests metric key

Notes:

  • Variant names must start with vllm or sglang
  • All metric keys within a variant are optional; unconfigured keys use built-in defaults
  • For homogeneous pools, set engine_type to vllm or sglang and configure variant keys in engines_metrics_keys; for heterogeneous pools, use auto (see Heterogeneous Engine Pool section)
  • The scheduler automatically detects which variant each member uses based on available metrics

Configuration Example

# Complete configuration example
global:
  interval: 5
  api_port: 8080
  api_host: 0.0.0.0
  log_level: INFO

f5:
  host: 192.168.1.100          # Required: F5 device address
  port: 443
  username: admin
  password_env: F5_PASSWORD

scheduler:
  pool_fetch_interval: 10
  metrics_fetch_interval: 3000

modes:
#Currently support s1 and s2. s1 use 2 metrics, s2 use 3 metrics
#You need test of them to see which one is better in your environment
  #- name: s1
    #w_a: 0.8 # In practice, w_a has greater impact on TTFT
    #w_b: 0.2
  - name: s2
    w_a: 0.4 # Weight for waiting queue metric
    w_b: 0.3 # Weight for cache usage metric  
    w_g: 0.3 # Weight for running requests metric

pools:
  - name: llm-pool-1           # Required: Pool name
    partition: Common
    engine_type: vllm          # Required: Engine type
    fallback:                  # Optional: Fallback configuration
      pool_fallback: false     # Pool-level fallback switch
      member_running_req_threshold: 25.0   # Exclude overloaded members
      member_waiting_queue_threshold: 20.0 # Exclude high-queue members
    metrics:
      schema: http
      path: /metrics
      timeout: 4
      APIkey: your-api-key
      metric_user: metrics_user
      metric_pwd_env: METRIC_PWD

  - name: llm-pool-2
    partition: tenant-1
    engine_type: sglang
    fallback:                  # Optional: Fallback configuration  
      pool_fallback: false     # Normal scheduling mode
      member_running_req_threshold: 30.0   # Higher threshold for SGLang
      # member_waiting_queue_threshold not set - no queue limit
    metrics:
      schema: https
      port: 9090               # Use unified metrics port
      path: /custom/metrics
      timeout: 5

# Engine variants metrics keys configuration (optional)
# Use this to support vLLM/SGLang variants with different metrics key names
engines_metrics_keys:
  vllm_ascend:                 # Huawei Ascend variant
    waiting_queue: vllm:num_requests_waiting
    cache_usage: vllm:kv_cache_usage_perc  # Ascend uses kv_cache instead of gpu_cache
    running_req: vllm:num_requests_running
  vllm_musa:                   # Moore Threads variant
    cache_usage: vllm:gpu_cache_usage_perc
  vllm-mlu:                    # Cambricon variant  
    cache_usage: vllm:mlu_cache_usage_perc

Algorithm Description

Please refer to [LLM-Inference-Gateway-Scheduler-Algorithm-Comparison-Analysis]

Weighted Random Selection

Weighted random selection based on each member's Score value:

  1. Calculate the sum of all members' Scores
  2. Generate a random number between 0 and the total sum
  3. Select the corresponding member based on which interval the random number falls into
  4. Members with higher Scores occupy larger intervals and have higher selection probability

Supported Metrics by Inference Engine

vLLM Engine (standard):

  • vllm:num_requests_waiting: Number of requests waiting in queue
  • vllm:gpu_cache_usage_perc: GPU cache usage percentage
  • vllm:num_requests_running: Number of requests currently running (for S2 algorithm)

SGLang Engine (standard):

  • sglang:num_queue_reqs: Number of requests in queue
  • sglang:token_usage: Token cache usage rate
  • sglang:num_running_reqs: Number of requests currently running (for S2 algorithm)

Engine Variants Support

For vLLM and SGLang variants (e.g., vllm_ascend for Huawei Ascend, vllm_musa for Moore Threads, vllm-mlu for Cambricon), the scheduler supports automatic metrics key detection and configuration.

Configuration: Use the engines_metrics_keys section to define custom metrics keys for each variant:

engines_metrics_keys:
  vllm_ascend:  # Huawei Ascend variant
    waiting_queue: vllm:num_requests_waiting
    cache_usage: vllm:kv_cache_usage_perc  # Different from standard vllm
    running_req: vllm:num_requests_running
  vllm_musa:    # Moore Threads variant
    cache_usage: vllm:gpu_cache_usage_perc
  vllm-mlu:     # Cambricon variant
    cache_usage: vllm:mlu_cache_usage_perc
  sglang_xxx:   # Custom SGLang variant
    waiting_queue: sglang:num_queue_reqs
    cache_usage: sglang:token_usage

Key Points:

  • Variant naming: Must start with vllm or sglang (underscore or hyphen allowed, e.g., vllm_ascend, vllm-mlu)
  • Pool engine_type: Use vllm or sglang for homogeneous pools; use auto for heterogeneous pools (see below). Do not use variant names as engine_type
  • Optional keys: Only configure keys that differ from the built-in standard; unconfigured keys fall back to built-in defaults
  • Priority: User-configured variant keys are tried first, then fall back to built-in keys
  • Auto-detection: The scheduler automatically detects which variant a member is using based on available metrics
  • Hot reload: Changes to engines_metrics_keys are hot-reloaded and take effect on the next metrics collection cycle

API Response: The /pools/{pool_name}/{partition}/status endpoint includes detected_variant, detected_engine_type (for auto pools), and detection_status (ok / partial / failed / degraded) for each member.

Heterogeneous Engine Pool (engine_type: auto)

When a single F5 Pool contains members running different Prometheus-based inference engines (e.g., vLLM and SGLang mixed together, including their variants), set engine_type: auto. The scheduler automatically detects each member's engine family (vLLM or SGLang) and variant from a single /metrics scrape—no per-member manual labeling is required.

Typical use cases:

  • One F5 Pool with both vLLM and SGLang backends serving the same model endpoint
  • Mixed standard and variant members (e.g., standard vLLM + Huawei Ascend vLLM + standard SGLang) in one pool

Configuration example:

pools:
  - name: pool_mixed_llm
    partition: Common
    engine_type: auto          # Heterogeneous pool: auto-detect vLLM / SGLang per member
    fallback:
      pool_fallback: false
    metrics:
      schema: http
      path: /metrics
      timeout: 4

# Optional: only needed when members use non-standard metric key names
engines_metrics_keys:
  vllm_v0_8:                 # Prefix vllm → vLLM candidate pool
    waiting_queue: vllm:pending_requests
    cache_usage: vllm:kv_cache_usage_perc
    running_req: vllm:active_requests
  sglang_v2:                 # Prefix sglang → SGLang candidate pool
    waiting_queue: sglang:pending_req
    cache_usage: sglang:token_usage_v2
    running_req: sglang:running_req_v2
  vllm_mindie:               # Variant without vllm: prefix (e.g., MindIE)
    waiting_queue: num_requests_waiting
    cache_usage: npu_cache_usage_perc
    running_req: num_requests_running

How auto-detection works:

  1. Stage 1 — Engine family: Scan Prometheus metrics for vLLM / SGLang signature keys (built-in + user-configured engines_metrics_keys). Each member is classified independently.
  2. Stage 2 — Variant keys: Within the detected family, match metric keys by priority (user variants first, then built-in defaults) and cache results for steady-state performance.

engine_type comparison:

Value Pool type Behavior
vllm Homogeneous Only vLLM metric keys are scanned; best cold-start performance
sglang Homogeneous Only SGLang metric keys are scanned
auto Heterogeneous Per-member vLLM / SGLang detection; standard members need no extra config

engines_metrics_keys rules in auto mode:

  • Use one top-level entry per variant; the variant name prefix (vllm_* / sglang_*) determines which engine family the keys belong to
  • Do not combine vLLM and SGLang keys under a single variant entry
  • Standard vLLM / SGLang members work without any variant configuration

detection_status values (visible in /pools/.../status):

Status Meaning
ok Engine family detected; required metrics (waiting_queue, cache_usage) collected
partial Engine family detected but a required metric is missing
failed Engine family not detected, or no usable metrics
degraded Previously cached keys temporarily unavailable; re-probing in progress

API response example (engine_type: auto):

{
  "name": "pool_mixed_llm",
  "partition": "Common",
  "engine_type": "auto",
  "member_count": 2,
  "members": [
    {
      "ip": "10.0.0.1",
      "port": 8001,
      "score": 0.72,
      "metrics": { "waiting_queue": 12.0, "cache_usage": 0.35, "running_req": 5.0 },
      "detected_variant": "vllm",
      "detected_engine_type": "vllm",
      "detection_status": "ok"
    },
    {
      "ip": "10.0.0.2",
      "port": 8010,
      "score": 0.85,
      "metrics": { "waiting_queue": 3.0, "cache_usage": 0.55, "running_req": 2.0 },
      "detected_variant": "sglang",
      "detected_engine_type": "sglang",
      "detection_status": "ok"
    }
  ]
}

Note: auto mode supports vLLM and SGLang (Prometheus /metrics) only. XInference uses a different metrics protocol and should be configured in a separate pool with engine_type: xinference.

Runtime Monitoring

Log Files

The scheduler generates detailed log files scheduler.log, including:

  • Configuration loading and hot reload records
  • Pool member fetch and update records
  • Metrics collection status and results
  • Score calculation process and results
  • API request and response records
  • Scheduling selection decision process
  • Error and exception information

Performance Metrics

Through API interfaces you can view:

  • Number and status of members in each Pool
  • Real-time Metrics data of members
  • Score distribution and trend changes
  • Selection result statistics and probability analysis
  • System runtime health status

Log Level Description

  • DEBUG: Shows all detailed information, including detailed process of each selection
  • INFO: Shows key operations and status changes
  • WARNING: Shows warning information such as missing configuration, connection issues, etc.
  • ERROR: Shows error information such as configuration errors, network failures, etc.
  • CRITICAL: Shows critical errors that may prevent the program from running

Troubleshooting

Common Issues

  1. F5 Connection Failure

    • Check F5 device network connectivity: ping <f5_host>
    • Verify username and password are correct
    • Confirm F5 device has iControl REST functionality enabled
    • Check if the user is locked as multi time failure logins
    • Change log level to debug to see detail

  2. Metrics Collection Failure

    • Check if inference engine services are running normally
    • Verify Metrics interface configuration is correct
    • Confirm network firewall settings allow access
    • Check inference engine's Metrics port and path
    • Change log level to debug to see detail

  3. Score Calculation Anomaly

    • Check if algorithm mode configuration is correct
    • Verify weight parameter settings (w_a + w_b recommended to equal 1)
    • Review Metrics data completeness
    • Confirm inference engine type configuration is correct
    • Change log level to debug to see detail

  4. Pool Member Fetch Failure

    • Verify Pool name and Partition match F5 configuration
    • Check Pool status on F5 device
    • Confirm F5 client connection and authentication are normal
    • Change log level to debug to see detail

Debug Mode

Enable detailed debug logging:

global:
  log_level: DEBUG

Or use backward compatible method:

global:
  log_debug: true

Health Check

Use health check interface to monitor service status:

curl http://localhost:8080/health

Normal response:

{"status": "healthy", "message": "Scheduler is running normally"}

Development Guide

Extending Support for New Inference Engines

For vLLM/SGLang Variants (Recommended):

Simply add the variant configuration to engines_metrics_keys in the config file:

engines_metrics_keys:
  vllm_custom:  # Your custom vLLM variant
    waiting_queue: vllm:custom_waiting_metric
    cache_usage: vllm:custom_cache_metric
    running_req: vllm:custom_running_metric

No code changes required. The scheduler will automatically detect and use the correct keys.

For Completely New Engine Types:

  1. Add new engine type in core/models.py:
class EngineType(Enum):
    VLLM = "vllm"
    SGLANG = "sglang"
    NEW_ENGINE = "new_engine"  # Add new engine
  1. Define key metrics in BASE_ENGINE_METRICS:
BASE_ENGINE_METRICS = {
    EngineType.NEW_ENGINE: {
        "waiting_queue": "new_engine:queue_length",
        "cache_usage": "new_engine:cache_usage",
        "running_req": "new_engine:running_requests"
    }
}
  1. Update parsing logic in metrics_collector.py (if metric format is different)

Implementing New Scheduling Algorithms

The project now supports two algorithms: S1 and S2. To implement additional algorithms:

  1. Add new mode in configuration:
modes:
  - name: s3
    w_a: 0.3
    w_b: 0.3
    w_g: 0.2
    w_h: 0.2  # Add new weight parameters as needed
  1. Add metrics support in core/models.py if new metrics are needed:
ENGINE_METRICS = {
    EngineType.VLLM: {
        "waiting_queue": "vllm:num_requests_waiting",
        "cache_usage": "vllm:gpu_cache_usage_perc",
        "running_req": "vllm:num_requests_running",
        "new_metric": "vllm:new_metric_name"  # Add new metric
    }
}
  1. Implement algorithm logic in core/score_calculator.py:
def _calculate_s3_scores(self, pool: Pool, mode_config: ModeConfig) -> None:
    # Implement S3 algorithm
    pass
  1. Update the main calculation method to support the new algorithm:
elif mode_config.name == "s3":
    self._calculate_s3_scores(pool, mode_config)

License

This project is for internal use, please comply with relevant usage terms.

About

repo for LLM inference LB scheduler and best practice

Resources

Readme

License

BSD-2-Clause license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity
Custom properties

Stars

4 stars

Watchers

0 watching

Forks

1 fork
Report repository

Releases 1

F5 BIG-IP Inference Gateway v1.1 Latest
Jul 24, 2025

Packages

 
 
 

Contributors

Languages