This is an experimental crawling project based on Google's 1998 architecture described in the paper "The anatomy of a large-scale hypertextual Web search engine".
My main goal is to learn about the process of crawling, parsing, indexing and ranking web pages. I'm looking into creating a very simple Ahrefs/SEMrush like application, so serving isn't the main focus.