A high-performance Python-based search engine designed to execute text queries over local file collections. The project documents an evolutionary engineering path: transitioning from baseline information retrieval library dependencies to a custom, zero-dependency Inverted Index implementation.
This project demonstrates core competencies in tokenization, information retrieval mechanics, index mapping structures, and algorithmic optimization.
The initial proof-of-concept leveraged standard parsing packages to handle document collection and search matching routines. This served as a baseline system to evaluate query accuracy and profile retrieval behavior.
To optimize performance and eliminate black-box runtime behavior, the engine was refactored into an optimized, pure-Python architecture built around an Inverted Index:
-
Mapping Framework: Documents are tokenized into normalized terms. The index maps each unique term to a posting list containing specific
Document IDswhere the term appears. -
$O(1)$ Average Lookup Complexity: Replaces slow sequential file scanning with instant hash-table matching using Python dictionaries. - Multi-Word Intersections: Implements custom parsing algorithms to support multi-word queries, resolving matches by evaluating the intersection of active posting lists.
- Data Normalization Pipe: Sanitizes input files by handling case folding and stripping punctuation to ensure high-fidelity term matching.
- Automated Directory Discovery: Scans an isolated local data directory, dynamically processing files and mapping them into memory.
- Multi-Word Search Capability: Resolves basic 1–2 word queries using optimized Boolean set logic across posting chains.