Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

383 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MLSDO - Group 03 - An ML System for Issue Classification

Our repository contains a production-style Machine Learning System which has been developed for the Machine Learning Systems Deployment and Operations (MLSDO) course.

The system implements a full ML lifecycle:

  • Data ingestion and preprocessing
  • Model training and evaluation
  • Experiment tracking and artifact storage
  • API-based inference
  • Frontend interaction
  • CI/CD automation
  • Deployment
  • Monitoring and logging

We aimed for a containerized and reproducible architecture.

System Overview

Core components:

  • Backend API: FastAPI-based inference and search service
  • Model Training: BERT-based multi-label classifier
  • Experiment Tracking: MLflow + PostgreSQL
  • Artifact Storage: MinIO (S3-compatible)
  • Data Versioning: DVC (Google Drive remote)
  • Frontend: Web UI for predictions and search
  • Monitoring & Logging: Prometheus, Grafana, Loki, Promtail
  • CI/CD: GitLab CI/CD with automated build, test, training, and deployment
  • Deployment Target: Remote VM via SSH + Docker Compose
├── backend/                # FastAPI backend
│   ├── api/
│   └── inference/ 
├── frontend/               # Frontend application
├── data/
│   ├── processing/         # Data extraction, preprocessing, validation
│   └── training/           # Training and evaluation logic
├── mlflow/                 # MLflow server + MinIO setup
├── grafana/                # Dashboards and provisioning
├── prometheus/             # Prometheus configuration
├── promtail/               # Log collection
├── loki/                   # Log aggregation
├── reports/                # Generated reports and outputs
├── tests/                  # Unit and integration tests
├── docker-compose.*.yml    # Service-specific Docker Compose files
├── dvc.yaml                # Reproducible data pipeline
├── environment.yaml        # Conda environment specification
├── requirements*.txt       # Python dependencies
├── pyproject.toml          # Formatting & linting configuration
└── .gitlab-ci.yml          # CI/CD pipeline definition

Manual Local Setup

Someone who wishes to setup the system locally, needs to execute all the docker compose files.

# Backend
docker compose -f docker-compose.backend.yml up -d

# MLflow + MinIO
docker compose -f docker-compose.mlflow.yml up -d

# MongoDB
docker compose -f docker-compose.mongo.yml up -d

# Monitoring
docker compose -f docker-compose.monitoring.yml up -d

# Frontend
docker compose -f docker-compose.frontend.yml up -d

They should also run training manually with one of the following commands:

# First install training dependencies
pip install -r requirements-ml.txt

# Quick test run with minimal data
python data/training/train.py --test  
# Train with test evaluation
python data/training/train.py
# Train on full data and save production model   
python data/training/train.py --full_training.  

The model is registered to MLflow. The API loads the model specified by MLFLOW_MODEL_URI (defaults to the latest alias).

CI/CD Pipeline Overview

The pipeline is defined in .gitlab-ci.yml.

The stages are shortly provided as follows:

  • lint: Enforces formatting (Black)
  • build: Builds and pushes Docker images
  • setup: Syncs repo and initializes VM
  • test: Runs toy-training and unit tests
  • update: Updates the data in the VM and reproduces the DVC pipeline
  • train: Executes scheduled training with evaluation, or full production training
  • deploy: Pulls images and redeploys services

Most jobs are configured to run on push events and are additionally restricted via rules:changes, so they execute only when the relevant files are modified. For the final submission pipeline, data updates and training with evaluation are intended to run on a scheduled basis (pending configuration in the GitLab UI). Full-data training and deployment are restricted to the main branch.

Below we also provide a full succesfull pipeline:

A succesfull CI/CD pipeline run (Part 1).

A succesfull CI/CD pipeline run (Part 2).

Testing and Linting Order

Linting and unit tests are executed before building Docker containers in our pipeline. The lint stage runs first and checks Python code formatting with Black, failing fast on style violations. This prevents wasting time on long container builds only to have the pipeline fail on a simple formatting issue. Integration tests (pytest stages) run after setup, once the required services (MLflow, backend) are running on the VM. This separation ensures quick feedback on code quality while reserving heavier integration testing for when the full environment is available. See .gitlab-ci.yml for the complete stage ordering.

Asynchronous Processing

Our API uses FastAPI's async/await for non-blocking I/O operations (database queries, HTTP handling). We also provide a batch endpoint that processes up to 100 issues in a single call, which is more efficient than individual predictions and helps handle peak loads. However, we acknowledge that the actual ML inference (BERT model) remains a blocking operation, while we use asyncio.run_in_executor() to move it to a thread pool, Python's Global Interpreter Lock (GIL) means this provides limited benefit for CPU-bound workloads. Due to time constraints, we did not implement true process-level decoupling using task queues such as Dramatiq, which would fully decouple prediction workloads from request handling by running inference in separate worker processes. See inference/service.py for the current async wrappers and routers/inference.py for the batch endpoint.

Push vs Pull Monitoring

We use a hybrid push and pull approach for monitoring, chosen deliberately based on each metric type's characteristics. Prometheus uses pull: it scrapes the /metrics endpoint exposed by our FastAPI backend every 15 seconds, which works well for application metrics since the backend is always running and accessible. Loki uses push via Promtail: container logs are actively pushed to Loki because logs are event-driven and shouldn't require Loki to poll every container. For training job metrics, we use the Pushgateway as an intermediary—training jobs (which are short-lived processes) push metrics to Pushgateway, and Prometheus pulls from there. This architecture ensures no metrics are lost from ephemeral jobs while maintaining efficient pull-based collection for long-running services. See prometheus/prometheus.yml, promtail/promtail-config.yml, and docker-compose.monitoring.yml.

Automation

Our CI/CD pipeline automates the entire ML lifecycle from code commit to production deployment. Data management is automated via DVC: the update stage pulls data from Google Drive, runs validation, and reproduces the preprocessing pipeline. Model training runs automatically in the train stage, logging experiments to MLflow with all metrics and artifacts. Deployment is fully automated—images are built, pushed to the GitLab Container Registry, and deployed to the VM via SSH without manual intervention. The only manual step is model selection through the MLflow UI, where we review metrics and promote a model to production by setting the alias to "production". Known limitation: Due to time constraints, the backend loads the model from MLflow only at startup and does not periodically check for newly promoted models. This means a backend restart is required after selecting a new model in MLflow—a design trade-off we acknowledge as technical debt that could be addressed with a model-refresh endpoint or periodic polling mechanism. See .gitlab-ci.yml and dvc.yaml for the pipeline configuration.

Prerequisites

The minimum prerequisites include:

  • Docker + Docker Compose v2
  • Python (if training locally)
  • .env The .env.example file is a template of how the actual .env file should look like. Make sure to fill in the secrets in the resulted .env. The .env file is not tracked by git, to avoid exposing any secrets.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages