Waterfall Lite - MAVLink Data Collection and Fault Detection System

A lightweight Python system for collecting MAVLink telemetry data from PX4 autopilots and performing real-time fault detection using neural network models. This system is designed for drone/UAV applications with defensive security and monitoring capabilities.

Overview

Waterfall Lite consists of several interconnected components that work together to:

• Stream MAVLink data from PX4 flight controllers at maximum rate
• Package data into uniform time batches for ML consumption
• Detect flight anomalies using trained neural networks
• Collect flight data for analysis and model training
• Inject parameter changes for flight controller reconfiguration

Architecture

Core Components

1. firehose.py - High-frequency MAVLink data streaming from PX4
2. uniform_pump.py - Time-batched data packaging for ML pipelines
3. data_collector.py - Flight data collection and storage
4. fault_detector.py - Real-time fault detection using neural networks
5. models.py - Neural network architectures (CNN and NN classifiers)
6. inject.py - Parameter injection and flight controller communication
7. px4_param_api.py - Direct C API interface for PX4 parameters
8. test.py - Testing and parameter manipulation utilities
9. waterfall_dashboard.py - Unified real-time visualization dashboard

Data Flow

PX4 Autopilot → firehose.py → uniform_pump.py → {data_collector.py, fault_detector.py}
                                                          ↓
                                                   models.py (fault detection)
                                                          ↓
                                                   inject.py (parameter correction)

Setup

Dependencies

Install using uv (recommended):

uv add pymavlink
uv add torch
uv add numpy
uv add rich
uv sync

Or with pip:

pip install pymavlink torch numpy rich rclpy

Hardware Requirements

• PX4-compatible flight controller (hardware or SITL)
• USB/serial connection to flight controller
• Python 3.8+ environment

Connection Setup

Hardware Connection

• Connect flight controller via USB serial
• Default connection: /dev/ttyACM0 at 2000000 baud
• Verify connection with: lsusb (should show flight controller)

SITL Connection

• Start PX4 SITL: make px4_sitl gazebo
• Default connection: udp:127.0.0.1:14550

Usage

Waterfall Dashboard (Recommended)

Launch the unified real-time visualization dashboard:

uv run python waterfall_dashboard.py

Features:

• Interactive Multi-Section Interface: Navigate with number keys [1-5]
  • [1] System Overview - High-level status of all components
  • [2] Firehose - Real-time MAVLink data streams with live attitude visualization
  • [3] Uniform Pump - Data batching pipeline with animated packet flow
  • [4] Fault Detector - Drone status monitoring with ASCII drone visualization
  • [5] Drone Visualization - Comprehensive flight data (attitude, position, sensors, battery, RC)

Dashboard Sections:

🔥 Firehose Section:

• Live data stream rates for all MAVLink message types
• Real-time attitude visualization with roll/pitch/yaw bars
• Historical sparklines for attitude trends
• Connection status and message throughput

⚡ Uniform Pump Section:

• Animated packet flow visualization showing data condensation
• Real-time batch progress countdown with progress bar
• Multi-step/single-step mode animations
• Bleeding domain statistics and recent batch history

🧠 Fault Detector Section:

• ASCII drone diagram showing motor status (●=OK, ✗=Failed)
• Real-time fault monitoring with expected fault types
• Fault history table with timestamps and severity
• Motor RPM visualization (simulated)

🚁 Drone Visualization Section:

• 3D attitude display with animated bars
• Top-down position map showing drone location
• Battery level with trend analysis
• RC input channel visualization
• Comprehensive sensor data (IMU, GPS, magnetometer)

Optional Model Loading:

uv run python waterfall_dashboard.py --model-path /path/to/fault_model.pth

Navigation:

• Number keys [1-5]: Switch between sections
• Q: Quit dashboard
• Real-time updates at 10Hz

Data Collection

Collect flight data during drone operations:

uv run python data_collector.py

What it does:

• Waits for drone takeoff
• Continuously collects sensor data during flight
• Saves timestamped data batches to data/ directory
• Automatically stops when drone lands

Configuration:

• Data directory: data/ (auto-created)
• Batch interval: Configurable in UniformPump
• Data format: JSON with timestamps

Fault Detection

Run real-time fault detection:

uv run python fault_detector.py --model-path /path/to/model.pth

What it does:

• Loads pre-trained neural network model
• Monitors flight data in real-time
• Detects anomalies and triggers alerts
• Initiates safety protocols when faults detected
• Reconfigures flight parameters automatically

Parameter Injection

Test parameter changes:

uv run python inject.py

What it does:

• Connects to flight controller via MAVLink
• Reads current PWM_MAIN_FUNC4 parameter
• Sets parameter to new value (default: 101)
• Verifies parameter change

Manual Testing

Interactive testing and parameter manipulation:

uv run python test.py

Root Files Description

Primary Entry Points

• data_collector.py: Main data collection application - run this during flights
• fault_detector.py: Real-time fault monitoring - run with trained models
• inject.py: Parameter injection utility - for testing parameter changes
• test.py: Development testing and debugging

Core Libraries

• firehose.py: MAVLink data streaming engine with terminal dashboard
• uniform_pump.py: Data batching and time synchronization
• models.py: Neural network model definitions (CNN, NN_classifier)
• px4_param_api.py: Low-level PX4 parameter API wrapper

Configuration

MAVLink Connection

Edit connection strings in main() functions:

# Hardware connection
firehose = MavFirehose(
    manager1,
    connection='serial:/dev/ttyACM0:2000000',
    serial_port='/dev/ttyACM0', 
    serial_baud=2000000,
)

# SITL connection
firehose = MavFirehose(
    manager1,
    connection='udp:127.0.0.1:14550'
)

Data Processing

Configure uniform_pump.py settings:

uniformPump = UniformPump(
    firehose,
    multi_step_count=5,        # Number of time steps
    condensation_mode='multi_step',  # 'raw', 'single_step', 'multi_step'
    batch_interval=1.0,        # Seconds between batches
    bleeding_domain_duration=15.0,  # Historical data window
)

Model Configuration

Neural network models in models.py:

• CNN: Convolutional neural network for image-like data (1x16x32 filters)
• NN_classifier: Fully connected network (86→256→256→64→3 neurons)
• Both support Monte Carlo dropout for uncertainty estimation

Data Format

Batch Structure

{
  "data": {
    "imu": [[accel_x, accel_y, accel_z, gyro_x, gyro_y, gyro_z], ...],
    "attitude": [[roll, pitch, yaw, rollspeed, pitchspeed, yawspeed], ...],
    "position": [[x, y, z], ...],
    "battery": [[voltage, current, remaining], ...]
  },
  "_batch_metadata": {
    "batch_number": 123,
    "condensation_mode": "multi_step",
    "total_messages_in_batch": 45
  }
}

Message Types

• imu: Accelerometer and gyroscope data
• attitude: Roll, pitch, yaw and rates
• position: Local NED coordinates
• velocity: Linear velocities
• battery: Power system status
• rc_channels: Radio control inputs
• servo_output: Actuator commands

Security Considerations

This system is designed for defensive security applications only:

• ✅ Flight anomaly detection
• ✅ Safety monitoring and alerts
• ✅ Data collection for analysis
• ✅ Parameter validation and correction
• ❌ No offensive capabilities
• ❌ No malicious parameter injection
• ❌ No unauthorized system access

Troubleshooting

Connection Issues

# Check USB connection
lsusb | grep -i px4

# Test serial connection
python -c "from pymavlink import mavutil; m = mavutil.mavlink_connection('/dev/ttyACM0', baud=2000000); print(m.wait_heartbeat())"

# Check permissions
sudo chmod 666 /dev/ttyACM0

Data Collection

• Ensure drone is armed and flying for data collection to start
• Check data/ directory permissions
• Monitor console output for connection status
• Verify MAVLink stream rates in firehose.py

Model Loading

• Verify model file path exists
• Check PyTorch model compatibility
• Ensure model architecture matches models.py definitions
• Validate input data dimensions (86 features for NN_classifier)

Development

Adding New Message Types

1. Add message handling in firehose.py _publish_typed_message()
2. Update data structures in uniform_pump.py
3. Modify condensation logic for new data format
4. Test with live MAVLink stream

Training New Models

1. Collect training data using data_collector.py
2. Process data according to model input requirements
3. Train models using PyTorch (external training pipeline)
4. Test models with fault_detector.py

Parameter API Extensions

1. Add new parameter types in px4_param_api.py
2. Extend encoding/decoding in inject.py
3. Test parameter changes in safe environment
4. Validate with PX4 parameter documentation

License

This project is intended for research, education, and defensive security applications. Use responsibly and in accordance with applicable regulations.