🔍 Semantic Search Engine on HDFS Logs — Big Data Project

A full Big Data pipeline that ingests, cleans, vectorizes, and semantically searches millions of HDFS log entries using Apache Spark, Sentence-Transformers, and PostgreSQL + pgvector.

---

📌 Project Overview

Modern distributed systems generate massive volumes of text logs. This project builds a complete platform capable of:

• Ingesting 11+ million raw HDFS log lines
• Cleaning and normalizing data at scale with Apache Spark
• Transforming log messages into 384-dimensional semantic vectors
• Storing and indexing vectors in PostgreSQL + pgvector
• Performing semantic search and pattern analysis on logs

Dataset: LogHub HDFS_v1 — 1.5 GB, 11,175,629 lines

---

🎯 Objectives

Objective	Description
Massive data processing	Handle 11M+ log lines with Apache Spark
Big Data architecture	Implement a full batch pipeline
Semantic search	Find similar logs without exact keyword matching
Vector database	Integrate pgvector into a Big Data workflow
Pattern detection	Cluster recurring error patterns automatically

---

🛠️ Tech Stack

Component	Version	Role
Python	3.11	Main language
Apache Spark (PySpark)	3.5.1	Massive ingestion & cleaning
Sentence-Transformers	latest	Embedding generation
PostgreSQL	18	Vector database
pgvector	latest	Vector similarity search
pandas / pyarrow	latest	Parquet file reading
scikit-learn	latest	KMeans clustering

---

📁 Project Structure

project_logs/
├── 01_exploration/
│   └── analyse_dataset.py            # Dataset analysis & stats
│
├── 02_spark/
│   └── nettoyage_spark.py            # Spark cleaning pipeline
│
├── 03_vectorisation/
│   ├── setup_db.py                   # PostgreSQL table creation
│   ├── vectorisation.py              # Embedding generation + insertion
│   └── create_index.py               # HNSW index creation
│
├── 04_recherche/
│   ├── cas1_logs_similaires.py       # Case 1: similar logs search
│   ├── cas2_groupes_erreurs.py       # Case 2: error clustering
│   └── cas3_evolution_temporelle.py  # Case 3: temporal analysis
│
├── rapport/                          # LaTeX technical report
├── .gitignore
├── requirements.txt
└── README.md

---

⚙️ Pipeline Architecture

HDFS.log (1.5 GB, 11M lines)
        │
        ▼
┌───────────────────┐
│   Apache Spark    │  → Parsing, cleaning, deduplication
│   (Phase 1 & 2)   │  → Output: 10,335,015 clean lines
└───────────────────┘
        │
        ▼
┌───────────────────┐
│ Sentence-         │  → all-MiniLM-L6-v2
│ Transformers      │  → 384-dim vectors, batch=256
└───────────────────┘
        │
        ▼
┌───────────────────┐
│ PostgreSQL        │  → 600,000 vectors inserted
│ + pgvector        │  → HNSW index (cosine similarity)
└───────────────────┘
        │
        ▼
┌───────────────────┐
│ Semantic Search   │  → 3 practical use cases
│ & Analytics       │
└───────────────────┘

---

🚀 Getting Started

Prerequisites

• Python 3.11 (via Anaconda)
• Java 11+ (required for Spark)
• PostgreSQL 18 with pgvector extension
• Windows: winutils.exe for Spark

1. Clone the repository

git clone https://github.com/ikramchouider/Project_Logs.git
cd Project_Logs

2. Create conda environment

conda create -n projet_logs python=3.11 -y
conda activate projet_logs

3. Install dependencies

pip install -r requirements.txt

4. Download the dataset

Download HDFS_v1 from LogHub and place it at:

data/HDFS_V1/HDFS.log
data/HDFS_V1/preprocessed/

5. Run the pipeline

# Step 1 — Explore the dataset
python 01_exploration/analyse_dataset.py

# Step 2 — Clean with Spark
python 02_spark/nettoyage_spark.py

# Step 3 — Setup database
python 03_vectorisation/setup_db.py

# Step 4 — Generate embeddings
python 03_vectorisation/vectorisation.py

# Step 5 — Create HNSW index (run directly in psql)
# SET maintenance_work_mem = '512MB';
# CREATE INDEX idx_hdfs_embedding ON hdfs_logs USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);

# Step 6 — Run search cases
python 04_recherche/cas1_logs_similaires.py
python 04_recherche/cas2_groupes_erreurs.py
python 04_recherche/cas3_evolution_temporelle.py

---

📊 Results Summary

Dataset stats

Metric	Value
Raw lines	11,175,629
After cleaning	10,335,015
Removed (duplicates/empty)	840,614 (7.5%)
INFO logs	10,015,984 (96.9%)
WARN logs	319,031 (3.1%)
Vectors inserted	600,000
Vector dimensions	384

Semantic search vs keyword search

Criteria	Keywords	Semantic
Results on natural queries	0 / 3	3 / 3
Average response time	0.216s	0.016s
Automatic anomaly grouping	✗	✓
Synonym robustness	✗	✓

---

📋 Practical Use Cases

Case 1 — Find similar logs to a given error

Retrieve all logs semantically similar to a query like "error receiving block connection refused" — even when exact words don't match.

Case 2 — Identify recurring error groups

Apply KMeans clustering (k=8) on 5,000 random vectors to automatically discover semantic groups. One cluster isolates exception logs without any manual rules.

Case 3 — Temporal evolution of errors

Track how similar error logs are distributed over time. Detected bursts of replication failures at specific hours, indicating MapReduce job saturation patterns.

---

⚠️ Notes

• The data/ folder is excluded from this repository (too large for GitHub). Download the dataset manually from LogHub.
• Spark on Windows requires winutils.exe — see setup instructions above.
• HNSW index was preferred over IVFFlat for faster construction on local hardware.

---

👩‍💻 Authors

Name	Group
CHOUIDER Ikram	SIQ2
DJOGHLAL Romaisa	SIQ2

Supervisor: Ens. Amrouche Karima
Module: Bases de Données Avancées — ESI 2024/2025