Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

265 Commits

Folders and files

Repository files navigation

SignalStack

AI-powered news intelligence and retrieval platform built with FastAPI, PostgreSQL, machine learning, semantic embeddings, and hybrid retrieval workflows.

SignalStack ingests RSS news feeds, normalizes and enriches article data, stores semantic embeddings locally, separates training labels from model predictions, and exposes operational APIs plus a lightweight frontend dashboard for search, ingestion, retrieval, and AI/ML demonstration workflows.

Demo

Video walkthrough: watch the SignalStack demo

Frontend dashboard:

http://127.0.0.1:8000/demo

Interactive API docs:

http://127.0.0.1:8000/docs

Preview

SignalStack Dashboard

Portfolio Purpose

SignalStack was built as a portfolio project to demonstrate backend software engineering, database-backed search, AI/ML enrichment workflows, semantic retrieval, and operational dashboard design.

The project is intended to support applications for entry-level software engineering, backend engineering, AI software engineering, AI platform engineering, and database-heavy application development roles.

What This Project Demonstrates

  • Backend API development with FastAPI
  • PostgreSQL-backed application design
  • Database migrations with Alembic
  • RSS ingestion and data normalization
  • Full-text, semantic, and hybrid retrieval workflows
  • Applied machine learning classification with scikit-learn
  • Local sentence-transformer embedding generation
  • Clear separation between ML training labels and inference predictions
  • Lightweight frontend dashboard development
  • Operational visibility for ingestion and AI workflows

Current Status

SignalStack is currently portfolio-demo ready.

Implemented functionality includes RSS ingestion, article storage, API search/filtering, full-text search, semantic search, hybrid retrieval, ML classification, embedding persistence, frontend dashboard views, and operational RSS feed controls.

The roadmap below describes possible future enhancements rather than required functionality for the current demo.


Features

News Ingestion

  • RSS feed ingestion pipeline
  • Configurable feed source management
  • Duplicate detection
  • Article normalization and text cleaning
  • RSS ingestion timeout handling
  • Operational ingestion dashboard
  • Feed synchronization between config and database
  • Feed health/error visibility
  • Live ingestion triggering from frontend

Search & Retrieval

  • PostgreSQL TSVECTOR full-text search
  • Weighted ranking with ts_rank_cd
  • Phrase search
  • Keyword filtering
  • Metadata filtering
  • Semantic embedding search
  • Related article retrieval
  • Hybrid keyword + semantic ranking
  • Pagination and sorting

AI & Machine Learning

  • Local/offline sentence-transformer embeddings
  • scikit-learn article classification
  • TF-IDF vectorization
  • Naive Bayes classification
  • Weak-supervision training-label workflow
  • Separation of training labels from model predictions
  • ML confidence scoring
  • Persisted vectorizers and model artifacts
  • Live inference API
  • Article similarity retrieval
  • Embedding persistence in PostgreSQL

Frontend Dashboard

  • Static frontend served by FastAPI
  • Dashboard statistics cards
  • Sticky operational navigation
  • RSS feed operational modal
  • Live ingestion controls
  • RSS synchronization status messages
  • Semantic search interface
  • Hybrid search interface
  • ML prediction testing
  • Related article lookup
  • Live dashboard refresh after ingestion

Operational Features

  • RSS feed source synchronization
  • Ingestion metrics and visibility
  • Feed activation/deactivation
  • Last ingest timestamps
  • Error propagation to frontend
  • Offline-capable AI architecture
  • Local ML artifact loading
  • Service-layer organization
  • Dockerized PostgreSQL deployment

Technology Stack

Backend

  • Python 3.13
  • FastAPI
  • SQLAlchemy ORM
  • Alembic
  • PostgreSQL 16
  • Docker

Machine Learning & AI

  • scikit-learn
  • sentence-transformers
  • semantic embeddings
  • cosine similarity retrieval
  • TF-IDF vectorization
  • Naive Bayes classification
  • weak-supervision labeling

Frontend

  • HTML
  • CSS
  • JavaScript

Architecture Overview

High-Level Platform Architecture

SignalStack Architecture

SignalStack follows a modular service-layer architecture designed for AI-enhanced retrieval, operational ingestion workflows, and offline-capable ML pipelines.

RSS Feeds
→ Ingestion Pipeline
→ Article Cleaning & Enrichment
→ Weak-Supervision Training Labels
→ ML Classification Inference
→ Semantic Embedding Generation
→ PostgreSQL Storage
→ Search / Retrieval APIs
→ Frontend Dashboard

RSS Ingestion Flow

RSS Ingestion Flow

The RSS ingestion pipeline synchronizes configured feed sources, performs ingestion, enriches article content with ML classification and embeddings, and updates operational dashboard statistics.

AI Retrieval Flow

AI Retrieval Flow

SignalStack supports keyword search, semantic retrieval, and hybrid ranking by combining PostgreSQL full-text search with local embedding similarity workflows.

ML Training and Inference Flow

ML Training and Inference Flow

SignalStack separates weak-supervision training labels from model prediction outputs. Training labels are used to generate classifier artifacts, while inference uses those artifacts to produce ml_category and ml_confidence values.

Core architectural goals:

  • offline-capable AI workflows
  • reusable service-layer logic
  • modular API endpoints
  • explainable retrieval systems
  • clear training vs inference separation
  • incremental platform evolution
  • lightweight deployment footprint

Frontend Demo Dashboard

SignalStack includes a lightweight frontend dashboard served directly by FastAPI.

Frontend URL:

http://127.0.0.1:8000/demo

The dashboard supports:

  • viewing live platform statistics
  • triggering RSS ingestion
  • viewing RSS feed synchronization status
  • keyword article search
  • semantic embedding search
  • hybrid retrieval search
  • ML prediction testing
  • RSS feed operational visibility
  • related article lookup

The frontend intentionally uses plain HTML, CSS, and JavaScript to keep the project lightweight and focused on backend engineering, AI retrieval systems, and operational workflows.

Dashboard Overview

SignalStack Dashboard

Platform Architecture & Status

SignalStack About and Status

RSS Feed Operations

RSS Feed Modal

Semantic Search

Semantic Search

Hybrid Retrieval

Hybrid Retrieval

ML Prediction

ML Prediction

ML Training and Inference Architecture

ML Training and Inference Flow


API Examples

Articles

GET /api/v1/articles

GET /api/v1/articles?search=ai

GET /api/v1/articles?phrase_search=artificial intelligence

GET /api/v1/articles?min_ml_confidence=0.70

Semantic Search

GET /api/v1/articles/semantic-search?query=artificial%20intelligence%20semiconductor%20chips

Hybrid Search

GET /api/v1/articles/hybrid-search?query=cybersecurity%20ransomware%20hospital

Related Articles

GET /api/v1/articles/{article_id}/similar

Machine Learning

POST /api/v1/ml/predict

GET /api/v1/ml/health

RSS Feed Operations

GET /api/v1/rss-feeds

POST /api/v1/rss-feeds/sync

POST /api/v1/rss-feeds/ingest

Dashboard

GET /api/v1/dashboard/stats


Example Dashboard Statistics

  • Total articles
  • Active RSS feeds
  • ML-classified articles
  • Embedded articles
  • Last article ingest timestamp
  • Last feed ingest timestamp

Project Structure

app/
├── api/
├── config/
├── crud/
├── db/
├── ingestion/
├── ml/
├── models/
├── schemas/
├── services/
├── static/
└── utils/

models/
└── ml/
    ├── article_classifier.joblib
    └── article_vectorizer.joblib

requirements/
└── requirements.txt

docs/
├── diagrams/
├── screenshots/
└── srs/

Demo Workflow

1. Clone the Repository

git clone <your-repository-url>
cd signalstack

2. Create a Python Virtual Environment

python -m venv .venv
source .venv/bin/activate

3. Install Dependencies

pip install -r requirements/requirements.txt

4. Configure Environment Variables

cp .env.example .env

APP_NAME="SignalStack API" ENV=dev DATABASE_URL=postgresql+psycopg://signalstack_user:signalstack_password@localhost:5432/signalstack

5. Start PostgreSQL

docker compose up -d

6. Run Database Migrations

python -m alembic upgrade head

7. Run RSS Ingestion

python -m app.ingestion.fetch_rss

8. Backfill Training Labels

python -m app.ml.backfill_training_labels

9. Train Article Classifier

python -m app.ml.train_article_classifier

10. Backfill ML Predictions and Confidence Scores

python -m app.ml.backfill_ml_confidence

11. Start FastAPI

For normal demo/runtime use:

python -m uvicorn app.main:app

For active development with automatic reload:

python -m uvicorn app.main:app --reload

Note: --reload is useful during active code editing, but it can increase CPU usage on some Linux laptop environments because Uvicorn watches the filesystem for changes.

12. Open the Frontend Dashboard

http://127.0.0.1:8000/demo

13. Open Interactive API Docs

http://127.0.0.1:8000/docs

AI Architecture Notes

Training vs Inference

SignalStack separates machine learning training workflows from inference workflows.

Training means generating model artifacts from labeled article data.

training_label + article text
→ TF-IDF vectorizer
→ Naive Bayes classifier
→ persisted .joblib artifacts

Inference means loading trained artifacts and using them to make predictions on article text.

article text
→ vectorizer artifact
→ classifier artifact
→ ml_category + ml_confidence

This distinction prevents model predictions from being reused as training labels and helps avoid self-reinforcing feedback loops.

Training Labels

SignalStack stores independent training labels separately from model prediction outputs.

training_label

represents a weak-supervision label used for training.

training_label_source

describes how that label was generated.

Current training labels are generated using a deterministic rule-based weak-supervision classifier.

This supports a cleaner ML lifecycle:

rule-based weak supervision
→ training_label
→ classifier training
→ model artifacts
→ inference
→ ml_category + ml_confidence

ML Predictions

ml_category

represents the classifier prediction produced during inference.

ml_confidence

represents the model confidence score associated with that prediction.

These values are model outputs, not ground-truth labels.

ML Artifacts

SignalStack persists lightweight scikit-learn artifacts under:

models/ml/

Current artifacts:

models/ml/article_classifier.joblib
models/ml/article_vectorizer.joblib

These artifacts are committed to the repository because they are small enough for the current project and support reproducible local inference.

Larger model assets, such as sentence-transformer model files, are intentionally managed outside the repository when appropriate.


Semantic Embeddings

SignalStack uses locally hosted sentence-transformer models for semantic embedding generation.

Embeddings are currently stored in PostgreSQL JSON columns for portability and development simplicity.

Future upgrade path:

  • pgvector integration
  • approximate nearest-neighbor retrieval
  • distributed embedding pipelines
  • local model registry organization
  • GPU-accelerated inference support

Hybrid Retrieval

Hybrid search combines:

  • PostgreSQL full-text ranking
  • semantic embedding similarity
  • weighted scoring strategies

to improve search relevance and contextual retrieval quality.


Design Goals

  • lightweight deployment
  • explainable retrieval systems
  • offline-capable AI workflows
  • modular architecture
  • clear training vs inference separation
  • operational visibility
  • local reproducibility
  • incremental engineering evolution
  • recruiter/demo friendliness

Future Roadmap

  • pgvector integration
  • ANN retrieval optimization
  • async ingestion workers
  • streaming ingestion pipelines
  • article summarization
  • entity extraction
  • topic modeling
  • improved weak-supervision label coverage
  • human-reviewed training labels
  • model evaluation reports
  • model registry workflows
  • automated retraining workflows
  • authentication/admin controls
  • deployment automation
  • ingestion scheduling
  • frontend visual analytics

Documentation

Additional project documentation:

  • docs/diagrams/signalstack_overview.puml
  • docs/diagrams/rss_ingestion_flow.puml
  • docs/diagrams/ai_retrieval_flow.puml
  • docs/diagrams/ml_training_inference_flow.puml
  • docs/screenshots/
  • docs/srs/signalstack_srs_outline.md

License

MIT License

Portfolio and educational project.


Content Attribution Notice

RSS feed content, headlines, excerpts, and associated media remain the property of their respective publishers and content owners.

SignalStack stores and processes publicly available RSS metadata and article excerpts for educational, research, portfolio, and software engineering demonstration purposes.

This project is intended to demonstrate backend engineering, AI retrieval workflows, semantic search, machine learning enrichment pipelines, weak-supervision training workflows, and operational dashboard architecture.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages