An AI-driven boss encounter that studies how you fight—then changes its tactics, voice, and soundtrack around you.
A completed graduation-project tech demo by Group Chlorine
Most boss fights become easier once the player memorizes the script. Adaptive Boss Fight breaks that pattern.
This project is a 2D action-game encounter built around a closed behavioral feedback loop. The Unity client captures how the player moves and fights, a Python AI backend converts that telemetry into a 16-dimensional behavioral profile, and a PyTorch classifier identifies the player's dominant combat style. The result is fed back into the game to reshape the boss's abilities and decision-making—while generative narration and adaptive music reinforce the same interpretation through audio.
The result is not simply a boss with more health or damage. It is an opponent that becomes a more relevant counter to the person playing.
| Adaptive combat | Player modeling | Generative voice | Adaptive music |
|---|---|---|---|
| A phase-based finite-state boss expands its ability pool and combat decision space in response to the player. | Raw combat events and spatial snapshots become a 16-feature vector classified into four playstyle archetypes. | Gemini produces context-aware taunts and Pocket-TTS turns them into voice-conditioned audio without blocking combat. | MusicGen prepares phase- and playstyle-aware tracks, while Unity handles runtime selection, crossfades, event effects, and local fallback. |
Underneath those four features is a full Unity-to-Python integration: typed JSON contracts, batched telemetry, PostgreSQL persistence, model inference, generated-media caching, asynchronous audio delivery, and multiple layers of graceful degradation.
The architecture deliberately separates the real-time game loop from expensive AI work:
- Unity captures behavior. Attacks, hits, dashes, hooks, healing, damage, wall actions, positions, health, and timing data are recorded during combat.
- Telemetry is batched. Periodic spatial snapshots and event records are buffered and sent outside the frame-critical path instead of issuing a network request every frame.
- FastAPI validates and routes. Pydantic request models enforce the C# ↔ Python data contract before data reaches a service or the database.
- PostgreSQL preserves the experiment trail. Players, sessions, combat sequences, events, snapshots, and extracted features remain traceable across the encounter.
- The classifier interprets the player. A PyTorch multilayer perceptron returns probabilities for four playstyles and exposes both the primary and secondary tendencies.
- Unity adapts the experience. Boss ability pools, selection behavior, narration, and music are updated through independent coordinators with safe local fallbacks.
This division keeps Unity responsive while allowing each AI subsystem to evolve independently.
The system does not classify a player from a single score or outcome. It describes how the player fights using 16 normalized features:
| Spatial behavior | Combat behavior | Mobility and survival |
|---|---|---|
| Average distance to boss | Attacks per minute | Air-time ratio |
| Danger-zone ratio | Charged attacks per minute | Dashes per minute |
| Movement-toward-boss ratio | Ranged attacks per minute | Hooks per minute |
| Average movement speed | Attack accuracy | Wall-slide/jump rate |
| Hook hit rate | Hit-taken rate | |
| Charged-to-regular ratio | Heal rate |
These features feed a compact 16 → 64 → 32 → 4 neural network with ReLU activations. The trained model, label mapping, and MinMax scaler are versioned with the repository so the exact inference pipeline can be reproduced.
The four output classes are:
- Aggressive — closes distance and maintains high offensive pressure.
- Defensive — prioritizes survival, spacing, and controlled engagement.
- Ranged/Tactical — creates distance and favors projectile-based control.
- Erratic — changes rhythm and position frequently, producing less predictable patterns.
Instead of discarding everything except the highest score, the runtime preserves the top-two probabilities. This makes hybrid behavior meaningful: the music system can distinguish Aggressive → Ranged/Tactical from Ranged/Tactical → Aggressive, and the gameplay layer can respond to mixed tendencies rather than forcing every player into a rigid box.
The boss is organized around a finite-state machine and a data-driven ability system:
- Four combat phases are synchronized with health thresholds and telemetry windows.
- Twelve concrete boss abilities inherit from a shared
BossAbilityScriptableObject contract. - Ability definitions expose tunable damage, cooldown, range, timing, and targeting data in the Unity Inspector.
- Archetype-aware selection allows the boss to favor counters that are relevant to the detected player style.
- Phase transitions update behavior without replacing or restarting the core combat controller.
This design keeps the adaptive layer extensible: adding a new move means implementing the ability contract and assigning data, not rewriting the boss brain.
The versioned training notebook records a balanced dataset of 204 labeled combat sequences:
- 163 training samples
- 41 held-out test samples
- 4 playstyle classes
- Class distribution: 51 Aggressive, 51 Defensive, 52 Erratic, 50 Ranged/Tactical
- Stratified 80/20 split with a fixed random seed of 42
- 200 training epochs
- 100% accuracy and 1.00 macro F1 on the held-out split
These results describe one held-out split of the project's controlled four-archetype dataset—not cross-validation or an unrestricted population of real-world players. The repository includes the notebook outputs, trained weights, scaler, label mapping, confusion matrix, and loss curve so the result is inspectable rather than presented as a black-box claim.
The narrator is a non-blocking dramatic layer driven by observed behavior changes:
Gameplay observation
↓
Context-rich prompt
↓
Gemini-generated taunt
↓
Pocket-TTS voice conditioning
↓
Cached WAV response
↓
Asynchronous Unity playback
The backend constrains generated lines to a short, direct boss persona, cleans the response, synthesizes it with a reusable reference-voice state, and stores the resulting WAV behind a deterministic cache key. Unity downloads the clip with UnityWebRequest and plays it without pausing the game loop.
The subsystem is deliberately resilient:
- Missing credentials switch text generation to deterministic, context-aware fallback lines.
- Failed remote narration switches Unity to a pool of 120 local narrator clips.
- Cached generated clips avoid repeating LLM and TTS work for the same combat context.
- Starting a new line stops prior narration cleanly instead of layering competing voices.
- Narration failure never blocks boss logic or phase progression.
Generating music synchronously in the middle of combat would trade immersion for latency. This project instead treats generation and playback as two separate engineering problems.
The Python generation pipeline uses Hugging Face Transformers with MusicGen to build a deterministic asset plan from:
- four boss phases,
- four canonical playstyles,
- ordered primary/secondary style pairs,
- four variants per pool,
- phase-aware prompt templates,
- stable prompt seeds and file checksums.
The tracked manifest contains 148 planned 30-second tracks:
Phase 1: 1 opening style pair × 4 variants = 4
Phases 2–4: 3 phases × 12 ordered style pairs × 4 variants = 144
───
Total 148
Generated WAV files are intentionally excluded from Git because they are large derived assets. The manifest, prompt system, coverage audit, resumable generation command, and asset checksums remain versioned, allowing the library to be regenerated on the target machine.
FastAPI mounts the generated asset directory under /music/assets. The music service resolves a manifest entry into an asset URL, and Unity downloads the selected WAV asynchronously before passing it to the playback manager.
At runtime, the music layer is more than a playlist:
- The classifier's ordered top-two styles select a phase-specific track pool.
- Double-buffered
AudioSourceplayback crossfades between full tracks. - Boss abilities, low-health states, and behavior shifts apply short event-driven audio effects.
- Next-track requests are serialized so overlapping HTTP calls cannot race.
- Revision numbers ensure a stale response cannot overwrite a newer phase transition.
- Missing generated assets immediately activate one of 19 bundled local combat tracks.
The hard part is coordination: classification, phase changes, track rotation, network latency, and match lifecycle events can all happen close together. The coordinator resolves those competing events explicitly instead of relying on timing luck.
The AI systems enhance the encounter, but none of them is allowed to become a single point of failure.
| Failure mode | Runtime behavior |
|---|---|
| Backend unavailable | Unity continues with local boss, music, and narrator behavior. |
| Generated music asset missing | The remote path is stopped and a local style-aware pool takes over. |
| Gemini key absent or generation fails | Deterministic contextual narration text is used. |
| Pocket-TTS or generated audio unavailable | Unity selects a bundled narrator clip. |
| Malformed JSON response | Failure callbacks release request state instead of leaving the coordinator locked. |
| Match ends during an asynchronous operation | Lifecycle revisions invalidate late coroutines before they can restart a stopped system. |
| Phase update collides with track rotation | The newer phase request supersedes the stale same-pool rotation. |
This fallback-first design lets the project demonstrate ambitious AI integration without sacrificing the reliability of the core game.
The PostgreSQL schema follows the lifetime of a play session:
Player
└── Game Session
└── Combat Sequence
├── Game Events (JSONB payloads)
├── Telemetry Snapshots (JSONB positions)
└── Feature Vector (16 behavioral metrics)
Relational keys preserve session integrity, while JSONB keeps heterogeneous event and position payloads flexible. A psycopg2 connection pool supports repeated game requests, and FastAPI routers isolate logging, classification, narration, and music behind separate service boundaries.
The following results were recorded in the project's documented validation environment and are detailed in the Final Report:
| Measurement | Recorded result |
|---|---|
| Combat frame rate | Stable 60 FPS |
| Per-event telemetry logging | ~2 ms |
| Feature-vector generation | 8 ms |
| Classification inference | 22 ms |
| Unity → Python → Unity adaptation round trip | 45 ms |
| Moderate backend load, 20 concurrent connections | ~137 requests/second, zero failures |
| High backend load, 100 concurrent connections | ~124 requests/second; 101/2,000 non-success responses |
The game measurements were recorded on Windows consumer hardware with 8–16 GB RAM and a discrete GPU. The FastAPI/PostgreSQL deployment ran on a Debian 12 Google Cloud Compute Engine VM, with ApacheBench executing from a second VM. The high-concurrency run exposed the expected ceiling of a small synchronous PostgreSQL connection pool: 101 of 2,000 requests returned non-success responses under 100 simultaneous connections. For a single-player tech demo this is outside normal operating load, but documenting it made the system's scaling boundary explicit.
| Layer | Technologies | Role |
|---|---|---|
| Game client | Unity 6000.0.58f1, C#, Physics2D, URP, Unity Input System, uGUI, TextMesh Pro | Rendering, input, combat, animation, UI, physics, orchestration |
| Game architecture | MonoBehaviour, ScriptableObject, finite-state machine, coroutines, UnityWebRequest | Data-driven abilities, lifecycle control, asynchronous API and audio work |
| Backend | Python 3.11, FastAPI, Uvicorn, Pydantic | Modular REST services, validation, inference and media delivery |
| Machine learning | PyTorch, scikit-learn, NumPy, Pandas, joblib, Jupyter | Feature preparation, MLP training, evaluation, serialization and inference |
| Generative narration | Google Gemini, Pocket-TTS, SciPy | Context-aware text, reference-conditioned speech and WAV output |
| Generative music | MusicGen, Hugging Face Transformers, PyTorch, SoundFile | Prompt-driven offline generation and reproducible audio manifests |
| Persistence | PostgreSQL, psycopg2, JSONB | Player/session hierarchy, event telemetry and extracted feature storage |
| Deployment and testing | Nginx, Google Cloud Compute Engine, ApacheBench | Reverse proxying, hosted backend and concurrent-load measurement |
Adaptive_Boss_Fight/
├── Adaptive_Boss_Fight_Unity/ # Unity game client and orchestration
│ ├── Assets/Scripts/ # Combat, telemetry, AI and audio integration
│ ├── Assets/BossAbilities/ # Data-driven ScriptableObject ability assets
│ └── Assets/Scenes/ArenaScene.unity # Reference playable encounter
├── Adaptive_Boss_Fight_No_Unity/
│ ├── api/ # FastAPI services and runtime logic
│ ├── db/schema.sql # PostgreSQL schema
│ ├── model_training/ # Dataset, notebooks and model artifacts
│ ├── generated_music/ # Versioned manifests; audio generated locally
│ └── generate_music_assets.py # Resumable MusicGen generation pipeline
├── Documents/ # Database diagrams and setup references
├── Reports/ # Design and final engineering reports
└── docs/assets/ # README architecture media
Local setup overview
- Unity
6000.0.58f1 - Python
3.11 - PostgreSQL
- A Python environment capable of installing PyTorch and Pocket-TTS
- Optional: a Gemini API key for generated narrator text
cd Adaptive_Boss_Fight_No_Unity
cp .env.example .env.local
python -m pip install -r api/requirements.txt./scripts/local_postgres.sh bootstrap
./scripts/run_backend_local.shThe backend is served at http://127.0.0.1:8000.
python -m pip install -r requirements-music.txt
./scripts/generate_small_music.sh playableGeneration is resumable, and the Unity-local music pools keep the encounter playable even when generated assets are not present.
Open Adaptive_Boss_Fight_Unity in Unity, load Assets/Scenes/ArenaScene.unity, and enter Play Mode.
For environment variables, asset generation modes, health probes, and runtime checks, see the complete Local AI Services Setup.
- Final Report — project objectives, implementation, evaluation, performance results, and engineering impact.
- Low-Level Design Report — package design, class interfaces, sequences, algorithms, traceability, and testing strategy.
- Database schema — runtime relational schema used by the backend.
- Database design diagram — editable database model.
- Local AI Services Setup — classifier, narrator, music, PostgreSQL, and Unity startup workflow.
Adaptive Boss Fight was designed and built as a collaborative graduation project by:
- Ali Berk Karaarslan
- Hüseyin Alper Koyuncu
- Nihat Emre Yüzügüldü
- Hüseyin Arda Tuz
The project brings together game programming, machine learning, backend engineering, databases, generative AI, audio systems, cloud deployment, and technical evaluation in one coherent real-time application.
